2024-03-25T16:02:57.138 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:02:57.138 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/orig.config.yaml 2024-03-25T16:02:57.185 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:02:57.276 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621240' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGhOWy56aoh5UtNOg3LmvNGF8b0GwQ4bUk7rw2EVB84w1QfE8UxSmJcQHHMGLXeWpLlkJ1tM67+kxnDCBO/sQI= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOvNpSlpcxMfV8QgjnjUklGa74zPD66ymf4Wxg95gUb/NB/Bhwumx03H/2i9zqOSV3uA5AUxbS8D+tIIdcOwm8= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:02:57.276 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:02:57.282 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:02:57.282 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:02:57.284 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:02:57.335 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:02:57.335 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:02:57.335 INFO:teuthology.packaging:ref: None 2024-03-25T16:02:57.335 INFO:teuthology.packaging:tag: None 2024-03-25T16:02:57.335 INFO:teuthology.packaging:branch: main 2024-03-25T16:02:57.336 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:02:57.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:02:57.573 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:02:57.575 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:02:57.582 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:02:57.582 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:02:57.597 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:02:57.620 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:02:57.627 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:02:57.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:48.630761', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGhOWy56aoh5UtNOg3LmvNGF8b0GwQ4bUk7rw2EVB84w1QfE8UxSmJcQHHMGLXeWpLlkJ1tM67+kxnDCBO/sQI='} 2024-03-25T16:02:57.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:48.632359', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOvNpSlpcxMfV8QgjnjUklGa74zPD66ymf4Wxg95gUb/NB/Bhwumx03H/2i9zqOSV3uA5AUxbS8D+tIIdcOwm8='} 2024-03-25T16:02:57.679 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:02:57.792 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:02:57.793 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:02:57.793 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:02:57.959 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0928f420d0>, signals=[15]) 2024-03-25T16:02:57.960 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:02:57.966 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:02:57.966 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T16:02:57.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:02:58.065 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T16:02:58.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:02:58.203 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:02:58.210 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-03-25T16:02:58.227 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-03-25T16:02:58.228 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:02:58.285 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:02:58.286 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:02:58.286 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:02:58.286 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:02:58.287 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-03-25T16:02:58.307 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-25T16:02:58.324 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-25T16:02:58.325 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-25T16:02:58.381 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-25T16:02:58.381 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-25T16:02:58.381 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-25T16:02:58.381 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:02:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:02:58.382 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-25T16:02:58.407 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:02:58.418 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:02:58.433 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:02:58.433 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:02:58.436 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:02:58.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:02:58.458 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:02:58.458 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:02:58.492 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:02:58.715 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:02:58.778 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:02:58.794 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:02:58.795 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:02:58.795 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:02:58.796 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:02:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:02:58.813 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-03-25T16:02:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:02:58.814 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-25T16:02:58.870 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:02:58.870 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-361.el9.x86_64 2024-03-25T16:02:58.870 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-03-25T16:02:58.872 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:02:58.872 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-03-25T16:02:58.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-03-25T16:03:01.115 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:03:01.290 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-25T16:03:02.890 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-25T16:03:03.074 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-25T16:03:03.814 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-25T16:03:04.006 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-25T16:03:05.632 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T16:03:05.731 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-25T16:03:09.673 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-25T16:03:11.291 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T16:03:13.866 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-25T16:03:17.034 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 788 kB/s | 6.1 MB 00:07 2024-03-25T16:03:18.693 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-25T16:03:18.791 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-25T16:03:20.881 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:20.882 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:20.919 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-25T16:03:20.973 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:03:20.984 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:03:20.984 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:20.984 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:03:20.984 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-03-25T16:03:20.985 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:03:20.986 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 76 M 2024-03-25T16:03:20.986 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 123 M 2024-03-25T16:03:20.986 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:03:21.409 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 116 kB/s | 44 kB 00:00 2024-03-25T16:03:26.002 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:03:28.076 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:28.076 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:28.171 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:03:28.183 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-03-25T16:03:28.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:03:28.185 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 76 M 2024-03-25T16:03:28.185 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-03-25T16:03:28.185 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:03:28.637 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 146 kB/s | 44 kB 00:00 2024-03-25T16:03:29.541 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.8 MB/s | 15 MB 00:08 2024-03-25T16:03:30.324 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 3.6 MB/s | 33 MB 00:09 2024-03-25T16:03:30.388 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.3 MB/s | 15 MB 00:02 2024-03-25T16:03:30.950 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.9 MB/s | 27 MB 00:09 2024-03-25T16:03:30.950 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:03:30.950 INFO:teuthology.orchestra.run.smithi096.stdout:Total 7.6 MB/s | 76 MB 00:09 2024-03-25T16:03:31.021 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T16:03:31.609 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:03:31.962 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:03:31.962 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:03:32.937 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.3 MB/s | 33 MB 00:04 2024-03-25T16:03:32.941 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:03:32.941 INFO:teuthology.orchestra.run.smithi203.stdout:Total 16 MB/s | 76 MB 00:04 2024-03-25T16:03:32.953 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:03:32.953 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:03:33.607 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:03:34.016 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:03:34.016 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:03:35.046 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:03:35.046 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:03:35.405 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:03:35.993 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:37.542 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:03:38.228 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:38.307 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:38.823 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:38.960 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:41.127 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:41.270 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:46.012 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:46.054 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:48.497 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:50.825 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:17.798 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:17.799 INFO:teuthology.orchestra.run.smithi096.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:17.799 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:04:17.843 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:18.261 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:18.261 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:18.261 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:18.261 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:04:18.674 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:04:19.000 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:19.136 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:19.137 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:19.137 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:19.137 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:19.137 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:19.137 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:04:19.137 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:19.137 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:19.216 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:19.217 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:19.217 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-03-25T16:04:19.257 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-03-25T16:04:19.257 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-03-25T16:04:19.257 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-03-25T16:04:19.258 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:04:19.259 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:19.260 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:19.260 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:19.260 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:19.881 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-03-25T16:04:19.947 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:19.947 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:19.947 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:04:19.991 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:04:20.765 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:04:21.088 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:21.194 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:21.194 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:21.194 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:21.194 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:21.194 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:04:21.194 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:21.194 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:21.274 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:21.274 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:21.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-03-25T16:04:21.278 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:04:21.319 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-03-25T16:04:21.319 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-03-25T16:04:21.319 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:21.320 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:04:21.321 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:21.322 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-03-25T16:04:21.323 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:21.323 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:21.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:21.324 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-03-25T16:04:21.339 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.30LH2aKHeq 2024-03-25T16:04:21.339 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.30LH2aKHeq 2024-03-25T16:04:21.411 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.30LH2aKHeq 2024-03-25T16:04:21.550 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.30LH2aKHeq is 6KB 2024-03-25T16:04:21.600 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.30LH2aKHeq 2024-03-25T16:04:21.615 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:04:21.680 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:04:21.681 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:04:21.681 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:04:21.681 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:04:21.683 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:04:21.829 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-03-25T16:04:21.907 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-03-25T16:04:23.307 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:04:23.343 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-03-25T16:04:23.346 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-25T16:04:23.361 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.j2sGP1M6l9 2024-03-25T16:04:23.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.j2sGP1M6l9 2024-03-25T16:04:23.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.j2sGP1M6l9 2024-03-25T16:04:23.566 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.j2sGP1M6l9 is 6KB 2024-03-25T16:04:23.614 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.j2sGP1M6l9 2024-03-25T16:04:23.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:04:23.703 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:04:23.703 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:04:23.703 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:04:23.703 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:04:23.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:04:23.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-03-25T16:04:51.861 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:04:51.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-25T16:04:51.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:04:53.853 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:04:53.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:04:53.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:28.479 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-25T16:05:37.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:05:37.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:51.872 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:05:55.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-25T16:06:00.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-25T16:06:00.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:04.311 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-03-25T16:06:04.704 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-25T16:06:04.704 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:04.704 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-03-25T16:06:04.719 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:04.719 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:04.719 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:04.719 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:05.721 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:06:05.722 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:05.786 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-03-25T16:06:05.831 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:07.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:06:07.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:08.364 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-25T16:06:08.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:06:08.877 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:08.878 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-25T16:06:08.892 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:08.892 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:08.893 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:08.893 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:09.893 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:06:09.894 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:10.003 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-03-25T16:06:10.053 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:10.053 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:06:10.061 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:06:10.061 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:10.063 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:10.083 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:06:10.115 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:06:10.150 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:06:10.150 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:10.155 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:10.210 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:06:10.219 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:06:10.219 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:10.233 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:10.266 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:10.279 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:10.283 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:10.312 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:10.316 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:06:10.323 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:06:10.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:10.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:10.370 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:06:10.379 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:06:10.379 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:10.418 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:10.438 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:10.537 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:10.603 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:10.689 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:10.735 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T16:06:10.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:10.805 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:10.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:10.897 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:10.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:11.022 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T16:06:11.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:11.090 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-03-25T16:06:11.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-25T16:06:11.181 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:11.252 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:11.669 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:06:11.679 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:06:11.679 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:06:11.718 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:06:11.770 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-03-25T16:06:11.838 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-03-25T16:06:11.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-25T16:06:11.887 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-25T16:06:11.889 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:06:11.889 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-03-25T16:06:11.910 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-03-25T16:06:11.910 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-25T16:06:11.927 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-25T16:06:11.928 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:06:11.928 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:11.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:12.009 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:06:12.010 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:12.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:12.112 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:06:12.122 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:06:12.129 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:06:12.279 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:06:12.296 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:06:12.298 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:10:26.530 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-25T16:10:26.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-25T16:10:26.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:26.651 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-03-25T16:10:26.738 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-25T16:10:26.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:10:26.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:26.839 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-25T16:10:26.921 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T16:10:26.922 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:10:26.935 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:10:26.936 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:10:26.937 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:10:26.940 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:10:26.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:10:26.972 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:10:26.989 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:10:26.993 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:10:27.010 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:10:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-03-25T16:10:27.035 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-03-25T16:10:27.039 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-25T16:10:27.051 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:10:27.052 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-25T16:10:27.067 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:10:27.069 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:10:27.086 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:10:27.127 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:10:27.130 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:10:27.130 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-03-25T16:10:27.130 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.131 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.131 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.131 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.137 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:10:27.141 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:10:27.151 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:10:27.151 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:10:27.151 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:10:27.151 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:10:27.158 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:10:27.158 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:10:27.158 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:10:27.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:27.160 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:10:27.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:27.310 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:10:27.310 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:10:27.314 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:10:27.314 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:10:27.421 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:10:27.421 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T16:10:27.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:10:27.449 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:10:27.449 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:10:27.509 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:10:27.510 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T16:10:27.510 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:10:27.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:10:27.538 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:10:27.538 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:10:27.598 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:10:27.606 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:10:27.665 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-25T16:10:27.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-25T16:10:27.688 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:10:27.755 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-03-25T16:10:27.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-03-25T16:10:27.902 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-25T16:10:27.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-03-25T16:10:27.989 INFO:teuthology.orchestra.run.smithi096.stdout:74 files removed 2024-03-25T16:10:28.015 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-03-25T16:10:28.640 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 200 kB/s | 82 kB 00:00 2024-03-25T16:10:28.866 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 148 kB/s | 82 kB 00:00 2024-03-25T16:10:29.148 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-25T16:10:29.292 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-25T16:10:29.699 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-25T16:10:29.709 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-25T16:10:30.419 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-25T16:10:30.502 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-03-25T16:10:32.003 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-25T16:10:32.044 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-25T16:10:32.770 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-25T16:10:32.914 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:10:34.752 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-25T16:10:38.922 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-25T16:10:40.537 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T16:10:43.022 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-03-25T16:10:45.318 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 19 MB 00:11 2024-03-25T16:10:47.955 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T16:10:50.409 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-25T16:10:50.412 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:10:50.413 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:10:50.414 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout:Install 30 Packages 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-03-25T16:10:50.415 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:10:50.416 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 58 M 2024-03-25T16:10:50.416 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:10:51.318 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-25T16:10:51.514 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 13 MB/s | 9.7 MB 00:00 2024-03-25T16:10:51.565 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 502 kB/s | 25 kB 00:00 2024-03-25T16:10:51.640 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 9.7 MB/s | 742 kB 00:00 2024-03-25T16:10:51.699 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 8.4 MB/s | 498 kB 00:00 2024-03-25T16:10:51.999 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-25T16:10:52.033 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.3 MB/s | 45 kB 00:00 2024-03-25T16:10:52.083 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-25T16:10:52.133 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-25T16:10:52.200 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-25T16:10:52.267 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-25T16:10:52.326 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-25T16:10:52.534 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 508 kB/s | 106 kB 00:00 2024-03-25T16:10:52.659 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-25T16:10:52.668 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-03-25T16:10:52.751 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 481 kB/s | 44 kB 00:00 2024-03-25T16:10:52.885 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 5.3 MB 00:02 2024-03-25T16:10:52.927 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-25T16:10:52.969 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T16:10:52.994 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 618 kB/s | 41 kB 00:00 2024-03-25T16:10:53.211 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 161 kB/s | 35 kB 00:00 2024-03-25T16:10:53.286 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 594 kB/s | 44 kB 00:00 2024-03-25T16:10:53.354 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 564 kB/s | 216 kB 00:00 2024-03-25T16:10:53.728 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 699 kB/s | 309 kB 00:00 2024-03-25T16:10:53.829 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-25T16:10:53.928 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T16:10:53.954 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-25T16:10:54.055 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-25T16:10:54.346 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 5.5 MB/s | 20 MB 00:03 2024-03-25T16:10:54.388 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-25T16:10:54.680 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 575 kB/s | 192 kB 00:00 2024-03-25T16:10:54.722 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-25T16:10:54.897 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-03-25T16:10:55.156 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 3.1 MB 00:00 2024-03-25T16:10:55.456 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.4 MB/s | 3.4 MB 00:00 2024-03-25T16:10:55.458 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:10:55.458 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-25T16:10:55.495 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:10:55.533 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:10:55.533 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:10:56.008 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:10:56.008 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:10:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:10:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:10:57.619 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:10:57.650 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:10:57.675 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:10:57.693 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:10:57.838 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:10:57.889 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:10:58.274 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:10:58.304 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:10:58.397 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:10:58.486 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:10:58.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:10:58.577 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:10:58.606 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:10:58.657 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:10:58.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:10:58.744 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-25T16:10:58.783 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:10:58.846 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:10:58.873 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:10:58.985 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:10:59.016 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:10:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:10:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:10:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:10:59.159 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:10:59.477 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:10:59.506 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:10:59.596 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:10:59.627 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:10:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:11:00.124 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:11:00.150 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:11:00.222 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:11:00.288 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:11:00.320 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:11:01.172 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options 2024-03-25T16:11:01.186 INFO:teuthology.orchestra.run.smithi096.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:11:01.187 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:11:01.188 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Install 30 Packages 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 58 M 2024-03-25T16:11:01.189 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:11:02.429 INFO:teuthology.orchestra.run.smithi096.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.0 MB/s | 5.3 MB 00:00 2024-03-25T16:11:02.456 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:02.470 INFO:teuthology.orchestra.run.smithi096.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 603 kB/s | 25 kB 00:00 2024-03-25T16:11:02.499 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:02.554 INFO:teuthology.orchestra.run.smithi096.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-25T16:11:02.613 INFO:teuthology.orchestra.run.smithi096.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 8.3 MB/s | 498 kB 00:00 2024-03-25T16:11:02.930 INFO:teuthology.orchestra.run.smithi096.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-25T16:11:02.972 INFO:teuthology.orchestra.run.smithi096.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-25T16:11:03.022 INFO:teuthology.orchestra.run.smithi096.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-25T16:11:03.046 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:03.080 INFO:teuthology.orchestra.run.smithi096.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-25T16:11:03.139 INFO:teuthology.orchestra.run.smithi096.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-25T16:11:03.140 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:11:03.197 INFO:teuthology.orchestra.run.smithi096.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-25T16:11:03.248 INFO:teuthology.orchestra.run.smithi096.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-25T16:11:03.269 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:11:03.270 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:11:03.270 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:11:03.270 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:03.303 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:03.457 INFO:teuthology.orchestra.run.smithi096.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-25T16:11:03.548 INFO:teuthology.orchestra.run.smithi096.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-25T16:11:03.615 INFO:teuthology.orchestra.run.smithi096.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-25T16:11:03.749 INFO:teuthology.orchestra.run.smithi096.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-25T16:11:03.816 INFO:teuthology.orchestra.run.smithi096.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-25T16:11:03.858 INFO:teuthology.orchestra.run.smithi096.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-25T16:11:04.150 INFO:teuthology.orchestra.run.smithi096.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 743 kB/s | 216 kB 00:00 2024-03-25T16:11:04.192 INFO:teuthology.orchestra.run.smithi096.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 838 kB/s | 35 kB 00:00 2024-03-25T16:11:04.243 INFO:teuthology.orchestra.run.smithi096.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-25T16:11:04.576 INFO:teuthology.orchestra.run.smithi096.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.5 MB/s | 9.7 MB 00:02 2024-03-25T16:11:05.159 INFO:teuthology.orchestra.run.smithi096.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 337 kB/s | 309 kB 00:00 2024-03-25T16:11:05.368 INFO:teuthology.orchestra.run.smithi096.stdout:(23/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 5.5 MB/s | 20 MB 00:03 2024-03-25T16:11:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-25T16:11:05.510 INFO:teuthology.orchestra.run.smithi096.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2024-03-25T16:11:05.843 INFO:teuthology.orchestra.run.smithi096.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-25T16:11:05.910 INFO:teuthology.orchestra.run.smithi096.stdout:(27/32): liboath-2.6.7-2.el9.x86_64.rpm 90 kB/s | 49 kB 00:00 2024-03-25T16:11:06.002 INFO:teuthology.orchestra.run.smithi096.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-25T16:11:06.069 INFO:teuthology.orchestra.run.smithi096.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-03-25T16:11:06.311 INFO:teuthology.orchestra.run.smithi096.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-25T16:11:06.503 INFO:teuthology.orchestra.run.smithi096.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 3.1 MB 00:00 2024-03-25T16:11:06.553 INFO:teuthology.orchestra.run.smithi096.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-03-25T16:11:06.560 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:06.560 INFO:teuthology.orchestra.run.smithi096.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-25T16:11:06.592 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:11:06.630 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:11:06.630 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:11:07.104 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:11:07.104 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:11:08.547 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:11:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:11:08.700 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:11:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:11:08.772 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:11:08.789 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:11:09.031 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:11:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:11:09.464 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:11:09.501 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:11:09.594 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:11:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:11:09.728 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:11:09.798 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:11:09.827 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:11:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:11:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:11:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:11:10.034 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:11:10.061 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:11:10.168 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:11:10.206 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:11:10.246 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:11:10.286 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:11:10.315 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:11:10.348 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:11:10.670 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:11:10.703 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:11:10.793 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:11:10.826 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:11:10.953 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:11:11.451 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:11:11.480 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:11:11.543 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:11:11.611 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:11:11.641 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:11:12.979 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:12.979 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-25T16:11:12.979 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-25T16:11:12.979 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-25T16:11:12.980 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-25T16:11:12.980 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-25T16:11:12.980 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-25T16:11:12.980 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-25T16:11:12.980 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:13.739 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:13.779 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:11:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:11:13.933 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:13.933 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:11:13.933 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:11:13.933 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:11:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:11:14.002 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:11:14.002 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:14.330 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:11:14.374 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:11:14.465 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:11:14.465 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:11:14.465 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:11:14.465 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:14.508 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:15.150 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:15.150 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:11:15.150 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:11:15.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:11:15.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:11:15.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:15.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:11:15.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:11:16.316 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:16.316 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.317 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.318 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:11:16.319 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:11:16.320 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:16.320 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:11:16.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-03-25T16:11:17.225 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:30 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:11:17.383 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:11:17.383 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:11:17.384 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 204 M 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:11:17.598 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:11:17.665 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-25T16:11:17.724 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-25T16:11:17.824 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-25T16:11:17.858 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 379 kB/s | 163 kB 00:00 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-03-25T16:11:24.001 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:24.911 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:11:24.941 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:24.943 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:11:25.039 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:11:25.039 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:26.093 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:11:26.094 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:11:26.095 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:11:26.096 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:11:26.097 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.341 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.342 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:11:27.343 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:11:27.344 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:11:27.344 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:27.344 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:11:27.693 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-03-25T16:11:28.248 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:30 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:11:28.394 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:11:28.395 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:11:28.396 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:28.396 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:11:28.396 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:11:28.396 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-03-25T16:11:28.396 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:28.397 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 48 M 2024-03-25T16:11:28.397 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 204 M 2024-03-25T16:11:28.397 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:11:28.870 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 601 kB/s | 217 kB 00:00 2024-03-25T16:11:29.004 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 300 kB 00:00 2024-03-25T16:11:29.079 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2024-03-25T16:11:29.304 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 205 kB/s | 163 kB 00:00 2024-03-25T16:11:30.266 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 158 kB/s | 187 kB 00:01 2024-03-25T16:11:35.749 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 47 MB 00:18 2024-03-25T16:11:35.751 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:35.751 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.6 MB/s | 48 MB 00:18 2024-03-25T16:11:35.766 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:11:35.787 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:11:35.787 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:11:36.169 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:11:36.169 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:11:36.741 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:11:36.806 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:11:36.851 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:11:36.884 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:11:36.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:11:36.960 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:11:40.961 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 3.8 MB/s | 47 MB 00:12 2024-03-25T16:11:40.964 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:40.964 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.8 MB/s | 48 MB 00:12 2024-03-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:11:40.996 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:11:40.996 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:11:41.374 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:11:41.375 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:11:41.474 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:11:41.552 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:11:41.966 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:11:42.045 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:11:42.095 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:11:42.130 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:11:42.170 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:11:42.198 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:11:43.982 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:11:43.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:11:43.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:11:43.982 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:11:43.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:11:43.983 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:11:44.390 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:11:44.391 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:11:44.391 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:44.391 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:11:44.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-03-25T16:11:45.274 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:11:45.432 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:11:45.437 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout:Install 36 Packages 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:45.439 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 30 M 2024-03-25T16:11:45.440 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 107 M 2024-03-25T16:11:45.440 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:11:45.963 INFO:teuthology.orchestra.run.smithi203.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-03-25T16:11:46.446 INFO:teuthology.orchestra.run.smithi203.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-25T16:11:46.648 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:11:46.739 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:11:47.221 INFO:teuthology.orchestra.run.smithi203.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-25T16:11:47.405 INFO:teuthology.orchestra.run.smithi203.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.3 MB/s | 247 kB 00:00 2024-03-25T16:11:47.572 INFO:teuthology.orchestra.run.smithi203.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-25T16:11:47.622 INFO:teuthology.orchestra.run.smithi203.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T16:11:47.649 INFO:teuthology.orchestra.run.smithi203.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:11:47.673 INFO:teuthology.orchestra.run.smithi203.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:11:47.706 INFO:teuthology.orchestra.run.smithi203.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:11:47.757 INFO:teuthology.orchestra.run.smithi203.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:11:47.849 INFO:teuthology.orchestra.run.smithi203.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-25T16:11:47.882 INFO:teuthology.orchestra.run.smithi203.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:11:47.933 INFO:teuthology.orchestra.run.smithi203.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:11:47.975 INFO:teuthology.orchestra.run.smithi203.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:11:48.183 INFO:teuthology.orchestra.run.smithi203.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 599 kB/s | 125 kB 00:00 2024-03-25T16:11:48.276 INFO:teuthology.orchestra.run.smithi203.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T16:11:48.376 INFO:teuthology.orchestra.run.smithi203.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-03-25T16:11:48.635 INFO:teuthology.orchestra.run.smithi203.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 7.7 MB/s | 17 MB 00:02 2024-03-25T16:11:48.660 INFO:teuthology.orchestra.run.smithi203.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-03-25T16:11:48.719 INFO:teuthology.orchestra.run.smithi203.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 341 kB/s | 151 kB 00:00 2024-03-25T16:11:48.785 INFO:teuthology.orchestra.run.smithi203.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-25T16:11:48.887 INFO:teuthology.orchestra.run.smithi203.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-25T16:11:48.911 INFO:teuthology.orchestra.run.smithi203.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-25T16:11:48.936 INFO:teuthology.orchestra.run.smithi203.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 570 kB/s | 172 kB 00:00 2024-03-25T16:11:48.987 INFO:teuthology.orchestra.run.smithi203.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-03-25T16:11:49.012 INFO:teuthology.orchestra.run.smithi203.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-25T16:11:49.037 INFO:teuthology.orchestra.run.smithi203.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-25T16:11:49.087 INFO:teuthology.orchestra.run.smithi203.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-25T16:11:49.112 INFO:teuthology.orchestra.run.smithi203.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T16:11:49.163 INFO:teuthology.orchestra.run.smithi203.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:11:49.180 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:11:49.188 INFO:teuthology.orchestra.run.smithi203.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-25T16:11:49.214 INFO:teuthology.orchestra.run.smithi203.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2024-03-25T16:11:49.247 INFO:teuthology.orchestra.run.smithi203.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-03-25T16:11:49.272 INFO:teuthology.orchestra.run.smithi203.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:11:49.323 INFO:teuthology.orchestra.run.smithi203.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-25T16:11:49.348 INFO:teuthology.orchestra.run.smithi203.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T16:11:49.350 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:49.351 INFO:teuthology.orchestra.run.smithi203.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-03-25T16:11:49.376 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:11:49.418 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:11:49.418 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:11:49.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:49.541 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:11:49.836 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-03-25T16:11:49.846 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:11:49.847 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:11:50.401 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:52 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:11:50.553 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:11:50.555 INFO:teuthology.orchestra.run.smithi096.stdout:========================================================================================= 2024-03-25T16:11:50.555 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:11:50.555 INFO:teuthology.orchestra.run.smithi096.stdout:========================================================================================= 2024-03-25T16:11:50.555 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:11:50.556 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:11:50.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout:========================================================================================= 2024-03-25T16:11:50.558 INFO:teuthology.orchestra.run.smithi096.stdout:Install 36 Packages 2024-03-25T16:11:50.559 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:11:50.559 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 30 M 2024-03-25T16:11:50.559 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 107 M 2024-03-25T16:11:50.559 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:11:51.205 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:11:51.267 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:11:51.310 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:11:51.337 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:11:51.390 INFO:teuthology.orchestra.run.smithi096.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 13 kB/s | 6.6 kB 00:00 2024-03-25T16:11:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:11:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:11:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:11:51.576 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:11:51.647 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:11:51.688 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:11:51.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:11:51.814 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:11:51.852 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:11:51.910 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:11:51.965 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:11:52.150 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:11:52.198 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:11:52.238 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:11:52.297 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:11:52.327 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:11:52.404 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:11:52.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:11:52.471 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:11:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:11:52.555 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:11:52.599 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:11:52.689 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:11:52.729 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:11:52.801 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:11:52.941 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:11:53.058 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:11:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:11:53.260 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:11:53.285 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:11:53.285 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:53.285 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:11:53.286 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:11:53.286 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:11:53.286 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:53.632 INFO:teuthology.orchestra.run.smithi096.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 787 kB/s | 2.1 MB 00:02 2024-03-25T16:11:54.728 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:11:54.754 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:54.773 INFO:teuthology.orchestra.run.smithi096.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 390 kB/s | 1.5 MB 00:03 2024-03-25T16:11:55.182 INFO:teuthology.orchestra.run.smithi096.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 606 kB/s | 247 kB 00:00 2024-03-25T16:11:55.204 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:11:55.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:55.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:11:55.440 INFO:teuthology.orchestra.run.smithi096.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 724 kB/s | 187 kB 00:00 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:11:55.462 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:55.527 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:11:55.532 INFO:teuthology.orchestra.run.smithi096.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2024-03-25T16:11:55.574 INFO:teuthology.orchestra.run.smithi096.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-25T16:11:55.624 INFO:teuthology.orchestra.run.smithi096.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-25T16:11:55.674 INFO:teuthology.orchestra.run.smithi096.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-25T16:11:55.767 INFO:teuthology.orchestra.run.smithi096.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-25T16:11:55.917 INFO:teuthology.orchestra.run.smithi096.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-03-25T16:11:55.975 INFO:teuthology.orchestra.run.smithi096.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-25T16:11:56.034 INFO:teuthology.orchestra.run.smithi096.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-25T16:11:56.101 INFO:teuthology.orchestra.run.smithi096.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-25T16:11:56.360 INFO:teuthology.orchestra.run.smithi096.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 483 kB/s | 125 kB 00:00 2024-03-25T16:11:56.518 INFO:teuthology.orchestra.run.smithi096.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-25T16:11:57.268 INFO:teuthology.orchestra.run.smithi096.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 201 kB/s | 151 kB 00:00 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:11:57.353 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:11:57.354 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:11:57.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:11:57.402 INFO:teuthology.orchestra.run.smithi096.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 326 kB/s | 43 kB 00:00 2024-03-25T16:11:57.769 INFO:teuthology.orchestra.run.smithi096.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 468 kB/s | 172 kB 00:00 2024-03-25T16:11:58.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:11:58.224 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:11:58.225 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:11:58.226 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:11:58.444 INFO:teuthology.orchestra.run.smithi096.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 531 kB/s | 358 kB 00:00 2024-03-25T16:11:58.578 INFO:teuthology.orchestra.run.smithi096.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-25T16:11:58.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-03-25T16:11:58.730 INFO:teuthology.orchestra.run.smithi096.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 117 kB/s | 18 kB 00:00 2024-03-25T16:11:58.853 INFO:teuthology.orchestra.run.smithi096.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 186 kB/s | 23 kB 00:00 2024-03-25T16:11:58.979 INFO:teuthology.orchestra.run.smithi096.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-25T16:11:59.112 INFO:teuthology.orchestra.run.smithi096.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-03-25T16:11:59.167 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:12 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:11:59.254 INFO:teuthology.orchestra.run.smithi096.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 327 kB/s | 46 kB 00:00 2024-03-25T16:11:59.264 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:11:59.317 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:11:59.318 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:11:59.318 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:11:59.371 INFO:teuthology.orchestra.run.smithi096.stdout:(27/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 601 kB/s | 4.7 MB 00:07 2024-03-25T16:11:59.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-03-25T16:11:59.480 INFO:teuthology.orchestra.run.smithi096.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 350 kB/s | 79 kB 00:00 2024-03-25T16:11:59.614 INFO:teuthology.orchestra.run.smithi096.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-03-25T16:11:59.797 INFO:teuthology.orchestra.run.smithi096.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 494 kB/s | 90 kB 00:00 2024-03-25T16:11:59.898 INFO:teuthology.orchestra.run.smithi096.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T16:11:59.969 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:12 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:00.122 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:00.124 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 747 k 2024-03-25T16:12:00.124 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 754 k 2024-03-25T16:12:00.124 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:00.206 INFO:teuthology.orchestra.run.smithi096.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 326 kB/s | 272 kB 00:00 2024-03-25T16:12:00.256 INFO:teuthology.orchestra.run.smithi096.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 640 kB/s | 230 kB 00:00 2024-03-25T16:12:00.373 INFO:teuthology.orchestra.run.smithi096.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-03-25T16:12:00.865 INFO:teuthology.orchestra.run.smithi096.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 648 kB/s | 427 kB 00:00 2024-03-25T16:12:01.547 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 525 kB/s | 747 kB 00:01 2024-03-25T16:12:01.547 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:01.548 INFO:teuthology.orchestra.run.smithi203.stdout:Total 525 kB/s | 747 kB 00:01 2024-03-25T16:12:01.548 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:01.554 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:01.555 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:01.560 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:01.561 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:01.648 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:02.128 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:02.374 INFO:teuthology.orchestra.run.smithi096.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 17 MB 00:08 2024-03-25T16:12:02.379 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:02.379 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.5 MB/s | 30 MB 00:11 2024-03-25T16:12:02.405 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:02.447 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:02.447 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:02.510 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:02.807 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:02.849 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:02.850 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:03.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:12:03.631 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:03.785 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:03.786 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:03.786 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:12:03.786 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 145 k 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 436 k 2024-03-25T16:12:03.787 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:04.022 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 621 kB/s | 145 kB 00:00 2024-03-25T16:12:04.022 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:04.022 INFO:teuthology.orchestra.run.smithi203.stdout:Total 617 kB/s | 145 kB 00:00 2024-03-25T16:12:04.023 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:04.030 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:04.030 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:04.083 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:04.084 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:04.166 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:04.234 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:12:04.273 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:12:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:04.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:12:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:04.314 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:04.314 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:12:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:04.381 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:12:04.427 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:12:04.459 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:12:04.550 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:12:04.637 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:12:04.679 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:12:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:12:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:12:04.843 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:12:04.891 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:12:04.947 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:12:05.130 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:12:05.181 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:12:05.211 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:12:05.246 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:12:05.277 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:12:05.343 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:05.349 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:05.378 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:12:05.412 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:12:05.460 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:12:05.496 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:12:05.561 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:12:05.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-03-25T16:12:05.651 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:12:05.686 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:12:05.757 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:12:05.901 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:12:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:12:06.178 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:06.186 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:12:06.217 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:06.242 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:06.242 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:06.242 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:12:06.242 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:06.243 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:06.243 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:06.275 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:06.329 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:06.330 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:06.330 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:06.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:12:06.983 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:07.135 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:12:07.136 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-25T16:12:07.137 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:07.138 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.9 M 2024-03-25T16:12:07.138 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-25T16:12:07.138 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:07.668 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:07.694 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:07.694 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:07.694 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:12:07.694 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:07.695 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:07.695 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:08.127 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:08.150 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:08.150 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:08.150 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:12:08.150 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:08.151 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:08.151 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:08.277 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 81 kB/s | 31 kB 00:00 2024-03-25T16:12:08.302 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 37 kB/s | 15 kB 00:00 2024-03-25T16:12:08.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:08.384 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:08.384 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:08.384 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:12:08.384 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:08.385 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:08.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:08.441 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:12:08.519 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-03-25T16:12:08.594 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-03-25T16:12:08.720 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.3 MB/s | 548 kB 00:00 2024-03-25T16:12:08.753 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 588 kB/s | 19 kB 00:00 2024-03-25T16:12:08.853 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-03-25T16:12:08.895 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-03-25T16:12:08.937 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.7 MB/s | 1.8 MB 00:01 2024-03-25T16:12:08.963 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 460 kB/s | 31 kB 00:00 2024-03-25T16:12:08.996 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.8 MB/s | 863 kB 00:00 2024-03-25T16:12:09.055 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:12:09.058 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:09.058 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-25T16:12:09.076 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:09.093 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:09.093 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:09.274 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:09.275 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:09.790 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:09.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:12:09.907 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:12:09.954 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:12:09.984 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:12:10.089 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:12:10.219 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:12:10.229 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:12:10.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:12:10.231 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:12:10.231 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:12:10.231 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:12:10.231 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:12:10.232 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:12:10.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:12:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:12:10.410 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:12:10.442 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:12:10.683 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:12:10.726 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:12:11.168 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:12:11.169 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:12:11.170 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:12:11.171 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:11.171 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:11.503 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-03-25T16:12:11.569 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:12:11.570 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:12:11.927 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:12:11.928 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:12:11.928 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:12:11.928 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:12:11.928 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:11.928 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:12.066 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:12.163 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:12.214 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:12.215 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:12:12.215 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:12.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:12:12.291 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-03-25T16:12:12.790 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:12.846 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:12.942 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:12.943 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T16:12:12.944 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-25T16:12:12.945 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:12.945 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-03-25T16:12:12.945 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-03-25T16:12:12.945 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:12.994 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:12.995 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:12.995 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 747 k 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 754 k 2024-03-25T16:12:12.996 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:13.298 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-25T16:12:13.299 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:13.299 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-25T16:12:13.299 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:13.304 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:13.304 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:13.310 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:13.310 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:13.380 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:13.481 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2024-03-25T16:12:13.514 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-25T16:12:13.590 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:12:13.615 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 344 kB/s | 34 kB 00:00 2024-03-25T16:12:13.782 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T16:12:13.831 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:13.849 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-25T16:12:13.875 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:12:13.894 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:14.294 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T16:12:14.384 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.1 MB/s | 7.4 MB 00:01 2024-03-25T16:12:14.525 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-25T16:12:14.543 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:14.543 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:14.543 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:14.543 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:14.543 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:14.544 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:14.550 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 754 kB/s | 195 kB 00:00 2024-03-25T16:12:14.576 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T16:12:14.578 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:14.579 INFO:teuthology.orchestra.run.smithi203.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-25T16:12:14.620 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:14.643 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:14.643 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:14.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:12:15.009 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:15.009 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:15.326 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:15.480 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:15.481 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:15.481 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:12:15.481 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:15.481 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:15.481 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 145 k 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 436 k 2024-03-25T16:12:15.482 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:15.714 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:15.788 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 475 kB/s | 145 kB 00:00 2024-03-25T16:12:15.789 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:15.789 INFO:teuthology.orchestra.run.smithi096.stdout:Total 473 kB/s | 145 kB 00:00 2024-03-25T16:12:15.789 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:15.796 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:15.796 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:15.799 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:12:15.830 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:12:15.848 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:15.848 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:12:15.959 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:12:15.986 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:16.072 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:16.072 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:16.072 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:12:16.072 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:16.137 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:12:16.174 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:12:16.255 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:12:16.927 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:12:17.037 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:12:17.043 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:17.044 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:17.044 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:17.044 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:17.044 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:17.044 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:17.283 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-03-25T16:12:17.856 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:17.952 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:18.004 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:18.005 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:12:18.005 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:18.077 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:12:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:12:18.665 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:20 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:18.818 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:18.818 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:12:18.819 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-03-25T16:12:18.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:18.821 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 6.9 M 2024-03-25T16:12:18.821 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-03-25T16:12:18.821 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:12:19.353 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:12:19.752 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T16:12:19.777 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 63 kB/s | 31 kB 00:00 2024-03-25T16:12:19.936 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-25T16:12:20.003 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-25T16:12:20.244 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-25T16:12:20.287 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2024-03-25T16:12:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-25T16:12:20.421 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-03-25T16:12:20.446 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.6 MB/s | 144 kB 00:00 2024-03-25T16:12:20.462 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 741 kB/s | 31 kB 00:00 2024-03-25T16:12:20.496 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-25T16:12:20.988 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.0 MB/s | 1.8 MB 00:01 2024-03-25T16:12:20.992 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:20.992 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-25T16:12:21.010 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:21.027 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:21.027 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:21.201 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:21.201 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:21.711 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:21.805 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:12:21.840 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:12:21.886 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:12:21.917 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:12:22.024 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:12:22.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:12:22.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:12:22.149 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:12:22.244 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:12:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:12:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:12:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:12:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:12:22.541 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:22.541 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:12:22.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:12:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:12:22.543 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:22.543 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:22.588 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:12:22.616 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:12:22.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:12:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:12:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:12:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:12:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:12:23.505 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:23.655 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:23.656 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:12:23.657 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:23.658 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:23.659 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-03-25T16:12:23.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:23.659 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-03-25T16:12:23.659 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-25T16:12:23.659 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:12:23.742 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:23.743 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:24.007 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:12:24.140 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2024-03-25T16:12:24.165 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-25T16:12:24.232 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T16:12:24.258 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T16:12:24.283 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:12:24.308 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-25T16:12:24.475 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 113 kB/s | 52 kB 00:00 2024-03-25T16:12:24.566 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:24.700 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2024-03-25T16:12:24.715 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:12:24.717 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-25T16:12:24.718 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-03-25T16:12:24.719 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:24.719 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 37 M 2024-03-25T16:12:24.719 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 209 M 2024-03-25T16:12:24.719 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:24.784 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 64 kB/s | 32 kB 00:00 2024-03-25T16:12:25.062 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 215 kB/s | 32 kB 00:00 2024-03-25T16:12:25.095 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-25T16:12:25.137 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 819 kB/s | 34 kB 00:00 2024-03-25T16:12:25.251 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 261 kB/s | 203 kB 00:00 2024-03-25T16:12:25.343 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 110 kB/s | 59 kB 00:00 2024-03-25T16:12:25.396 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-25T16:12:25.605 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-25T16:12:25.663 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 217 kB 00:00 2024-03-25T16:12:25.876 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 143 kB/s | 90 kB 00:00 2024-03-25T16:12:25.956 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-03-25T16:12:26.077 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 760 kB/s | 1.0 MB 00:01 2024-03-25T16:12:26.080 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:26.080 INFO:teuthology.orchestra.run.smithi203.stdout:Total 866 kB/s | 2.0 MB 00:02 2024-03-25T16:12:26.081 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:26.096 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:26.097 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:26.195 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:26.196 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:26.372 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T16:12:26.631 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 756 kB/s | 195 kB 00:00 2024-03-25T16:12:26.733 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:26.865 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 613 kB/s | 245 kB 00:00 2024-03-25T16:12:26.909 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:12:26.978 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:12:27.034 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:12:27.088 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:12:27.129 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:12:27.224 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:12:27.332 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:12:27.390 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:12:28.127 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:12:28.171 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:12:28.205 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:12:28.254 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:12:28.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:12:29.108 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:12:29.108 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:12:29.108 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:12:29.109 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:12:29.473 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:12:29.473 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:12:29.474 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:12:29.475 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:29.475 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:29.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:12:30.310 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:30.459 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:12:30.460 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-25T16:12:30.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:30.462 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-03-25T16:12:30.462 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-03-25T16:12:30.462 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:30.781 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.3 MB/s | 7.4 MB 00:05 2024-03-25T16:12:30.784 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:30.785 INFO:teuthology.orchestra.run.smithi096.stdout:Total 6.1 MB/s | 37 MB 00:06 2024-03-25T16:12:30.828 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:30.853 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:30.853 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:30.864 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.2 MB/s | 248 kB 00:00 2024-03-25T16:12:31.207 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:31.208 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:31.257 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:12:31.332 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 775 kB/s | 58 kB 00:00 2024-03-25T16:12:31.399 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 213 kB/s | 138 kB 00:00 2024-03-25T16:12:31.424 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-03-25T16:12:31.724 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 5.8 MB 00:00 2024-03-25T16:12:31.726 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:31.727 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2024-03-25T16:12:31.739 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:31.750 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:31.750 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:31.837 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:31.838 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:31.917 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:31.995 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:12:32.035 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:12:32.136 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:12:32.141 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:32.188 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:12:32.315 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:12:32.366 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:12:32.367 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:12:32.396 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:12:32.484 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:12:32.896 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:12:32.980 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:12:33.040 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:12:33.072 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:12:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:12:33.250 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:12:33.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:12:34.262 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:12:34.262 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:34.262 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:34.262 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:34.262 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:34.263 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:34.486 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:12:34.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-03-25T16:12:35.124 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:35.274 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:12:35.275 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:35.276 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 964 k 2024-03-25T16:12:35.277 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.8 M 2024-03-25T16:12:35.277 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 336 kB/s | 79 kB 00:00 2024-03-25T16:12:35.516 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:12:35.547 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:12:36.661 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 639 kB/s | 885 kB 00:01 2024-03-25T16:12:36.662 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:36.663 INFO:teuthology.orchestra.run.smithi203.stdout:Total 695 kB/s | 964 kB 00:01 2024-03-25T16:12:36.663 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:36.685 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:36.685 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:36.760 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:36.763 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:36.947 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:37.078 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:12:37.152 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:12:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:12:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:12:38.049 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:12:38.049 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:12:38.050 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:38.149 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:38.475 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:12:38.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:38.475 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:38.475 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:38.475 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:38.476 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:38.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-03-25T16:12:38.873 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:12:39.073 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:39.223 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:39.223 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T16:12:39.224 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:39.225 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 332 k 2024-03-25T16:12:39.225 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.5 M 2024-03-25T16:12:39.225 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:39.342 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 589 kB/s | 69 kB 00:00 2024-03-25T16:12:39.439 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:39.587 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:39.588 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:12:39.589 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-03-25T16:12:39.590 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:39.591 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.0 M 2024-03-25T16:12:39.591 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-03-25T16:12:39.591 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:39.767 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 487 kB/s | 264 kB 00:00 2024-03-25T16:12:39.768 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:39.768 INFO:teuthology.orchestra.run.smithi203.stdout:Total 611 kB/s | 332 kB 00:00 2024-03-25T16:12:39.769 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:39.777 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:39.777 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:39.817 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:39.817 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:39.905 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-03-25T16:12:39.931 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-25T16:12:39.964 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:39.981 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:12:40.006 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:12:40.031 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:12:40.057 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T16:12:40.157 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 150 kB/s | 15 kB 00:00 2024-03-25T16:12:40.158 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:12:40.182 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-25T16:12:40.195 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:12:40.207 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 122 kB/s | 52 kB 00:00 2024-03-25T16:12:40.211 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:12:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:12:40.211 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:40.299 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-25T16:12:40.333 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-25T16:12:40.358 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-25T16:12:40.467 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-25T16:12:40.468 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:40.469 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-25T16:12:40.469 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:40.483 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:40.483 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:40.577 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:40.579 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:40.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:12:41.096 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:41.269 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:12:41.282 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:12:41.282 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:41.282 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:41.283 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:41.283 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:12:41.283 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:41.283 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:41.325 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:12:41.382 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:12:41.419 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:12:41.468 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:12:41.561 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:12:41.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-03-25T16:12:41.662 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:12:41.712 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:12:42.225 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:42.382 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:42.382 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:42.382 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:12:42.382 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 127 k 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 456 k 2024-03-25T16:12:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:42.421 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:12:42.478 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:12:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:12:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:12:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:12:43.009 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 202 kB/s | 127 kB 00:00 2024-03-25T16:12:43.009 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:43.010 INFO:teuthology.orchestra.run.smithi203.stdout:Total 202 kB/s | 127 kB 00:00 2024-03-25T16:12:43.010 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:43.016 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:43.016 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:43.075 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:43.076 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:43.233 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:43.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:43.411 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:12:43.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:12:43.791 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:43.796 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:12:43.796 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:43.796 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:43.796 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:12:43.797 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:43.798 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:44.044 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:44.077 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:44.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-03-25T16:12:44.605 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:46 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:44.754 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:12:44.755 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 6.8 M 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-03-25T16:12:44.756 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:44.927 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:45.028 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:45.082 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:45.083 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:45.083 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:45.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-03-25T16:12:45.197 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:12:45.289 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 371 kB/s | 138 kB 00:00 2024-03-25T16:12:45.556 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 216 kB/s | 58 kB 00:00 2024-03-25T16:12:45.623 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:12:45.648 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 555 kB/s | 51 kB 00:00 2024-03-25T16:12:45.758 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:45.907 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:45.908 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:45.909 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:45.909 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:45.909 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 k 2024-03-25T16:12:45.909 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 140 k 2024-03-25T16:12:45.909 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:46.040 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 221 kB/s | 248 kB 00:01 2024-03-25T16:12:46.042 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:46.043 INFO:teuthology.orchestra.run.smithi096.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-25T16:12:46.057 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:46.070 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:46.070 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:46.159 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:46.159 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:46.189 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 111 kB/s | 31 kB 00:00 2024-03-25T16:12:46.189 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:46.189 INFO:teuthology.orchestra.run.smithi203.stdout:Total 111 kB/s | 31 kB 00:00 2024-03-25T16:12:46.190 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:46.193 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:46.193 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:46.226 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:46.226 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:46.371 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:46.449 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:46.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:46.629 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:12:46.674 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:12:46.880 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:47.151 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:12:47.298 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:12:47.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-03-25T16:12:47.363 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:12:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:12:47.900 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:47.997 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:48.049 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:48.050 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:48.050 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:48.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-03-25T16:12:48.191 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:12:48.191 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:12:48.191 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:12:48.192 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:12:48.192 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:12:48.192 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:12:48.497 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:12:48.497 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:48.673 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:01 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:48.769 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-03-25T16:12:48.771 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:48.823 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:48.824 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:48.824 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:48.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-03-25T16:12:49.334 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:49.457 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:49.483 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:49.484 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 964 k 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.8 M 2024-03-25T16:12:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:49.553 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:49.607 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:49.608 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:49.609 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:49.609 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 637 kB/s | 79 kB 00:00 2024-03-25T16:12:49.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-03-25T16:12:50.009 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 885 kB 00:00 2024-03-25T16:12:50.010 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:50.010 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-03-25T16:12:50.011 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:50.031 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:50.031 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:50.104 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:50.104 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:50.220 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:50.289 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:50.322 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:50.375 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:50.376 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:50.376 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:12:50.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-03-25T16:12:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:12:51.002 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:04 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:51.099 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:51.151 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:51.152 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:51.152 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:51.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-03-25T16:12:51.229 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:12:51.230 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:51.526 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:51.769 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:04 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:51.822 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-volume 2024-03-25T16:12:51.866 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:51.918 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:51.919 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:12:51.919 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:51.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-03-25T16:12:52.397 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:52.549 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:52.549 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:52.549 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 332 k 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.5 M 2024-03-25T16:12:52.550 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:52.673 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 561 kB/s | 69 kB 00:00 2024-03-25T16:12:52.696 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:52.697 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 k 2024-03-25T16:12:52.698 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 231 k 2024-03-25T16:12:52.698 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:52.823 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 966 kB/s | 264 kB 00:00 2024-03-25T16:12:52.824 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:52.825 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-25T16:12:52.825 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:52.834 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:52.834 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:52.873 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:52.874 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:52.917 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 388 kB/s | 85 kB 00:00 2024-03-25T16:12:52.918 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:52.918 INFO:teuthology.orchestra.run.smithi203.stdout:Total 385 kB/s | 85 kB 00:00 2024-03-25T16:12:52.918 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:52.926 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:52.926 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:52.980 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:52.980 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:53.023 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:53.114 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:53.205 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:12:53.215 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:53.242 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:12:53.259 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:12:53.259 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:53.259 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:12:53.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:53.624 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:53.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:53.952 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:53.952 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:53.952 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.952 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:53.952 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:53.999 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:12:54.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-03-25T16:12:54.328 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:12:54.328 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:54.328 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:54.328 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:54.328 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:12:54.329 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:54.329 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:54.639 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-03-25T16:12:54.770 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:54.923 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:54.925 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:54.925 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.1 M 2024-03-25T16:12:54.925 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 13 M 2024-03-25T16:12:54.925 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:55.216 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:55.364 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 127 k 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-03-25T16:12:55.365 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:55.830 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 272 kB/s | 127 kB 00:00 2024-03-25T16:12:55.831 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:55.831 INFO:teuthology.orchestra.run.smithi096.stdout:Total 272 kB/s | 127 kB 00:00 2024-03-25T16:12:55.831 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:55.836 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:55.836 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:55.892 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:55.892 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:55.913 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.1 MB/s | 3.1 MB 00:00 2024-03-25T16:12:55.914 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:55.914 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-25T16:12:55.914 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:55.925 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:55.925 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:56.002 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:56.002 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:56.037 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:56.145 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:56.433 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:56.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:56.521 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:56.521 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:56.522 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:12:56.522 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:12:56.522 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:12:56.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:56.593 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:56.894 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:57.118 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-03-25T16:12:57.690 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:57.717 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:57.718 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:57.718 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:12:57.718 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:57.718 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:57.718 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:12:57.791 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:57.845 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:57.846 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:12:57.846 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:12:57.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-03-25T16:12:57.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-03-25T16:12:58.494 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:12:58.578 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:12:58.643 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:12:58.643 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:58.643 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:12:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 31 k 2024-03-25T16:12:58.645 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 140 k 2024-03-25T16:12:58.645 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:12:58.725 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:12:58.726 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:12:58.727 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 171 k 2024-03-25T16:12:58.727 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 494 k 2024-03-25T16:12:58.727 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:12:59.109 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 67 kB/s | 31 kB 00:00 2024-03-25T16:12:59.110 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:59.110 INFO:teuthology.orchestra.run.smithi096.stdout:Total 67 kB/s | 31 kB 00:00 2024-03-25T16:12:59.111 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:12:59.114 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:12:59.114 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:12:59.143 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:12:59.144 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:12:59.269 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:12:59.366 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:59.435 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 242 kB/s | 171 kB 00:00 2024-03-25T16:12:59.435 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:59.436 INFO:teuthology.orchestra.run.smithi203.stdout:Total 242 kB/s | 171 kB 00:00 2024-03-25T16:12:59.436 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:12:59.444 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:12:59.444 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:12:59.502 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:12:59.502 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:12:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:12:59.750 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:12:59.759 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:00.010 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:00.169 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-03-25T16:13:00.211 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:00.490 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:00.491 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:00.491 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:13:00.491 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:00.491 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:00.491 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:13:00.732 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:00.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:13:00.828 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:00.879 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:00.937 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-03-25T16:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:13:01.470 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:13:01.471 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:01.472 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T16:13:01.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:01.472 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 91 M 2024-03-25T16:13:01.472 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 498 M 2024-03-25T16:13:01.472 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:13:01.497 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:01.646 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:01.646 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:01.646 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:01.693 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-03-25T16:13:02.255 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:04 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:02.350 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:02.402 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:02.403 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:02.403 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:02.449 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-03-25T16:13:03.010 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:03.106 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:03.157 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:03.158 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:03.158 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:03.209 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-03-25T16:13:03.770 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:03.865 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:03.917 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:03.918 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:03.918 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:03.970 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-03-25T16:13:04.533 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:06 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:04.629 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:04.680 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:04.681 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:13:04.681 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:04.732 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-03-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-03-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 85 k 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 231 k 2024-03-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 271 kB/s | 85 kB 00:00 2024-03-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi096.stdout:Total 270 kB/s | 85 kB 00:00 2024-03-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:06.766 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:06.975 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-03-25T16:13:07.546 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:09 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:07.699 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:07.699 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:07.700 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.1 M 2024-03-25T16:13:07.701 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 13 M 2024-03-25T16:13:07.701 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:13:08.256 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 3.3 MB/s | 22 MB 00:06 2024-03-25T16:13:08.348 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.8 MB/s | 3.1 MB 00:00 2024-03-25T16:13:08.349 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:08.349 INFO:teuthology.orchestra.run.smithi096.stdout:Total 4.8 MB/s | 3.1 MB 00:00 2024-03-25T16:13:08.349 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:13:08.358 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:13:08.358 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:13:08.431 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:13:08.431 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:13:08.862 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:13:08.933 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:08.957 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:08.957 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:08.957 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:13:08.957 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:08.957 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:08.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:10.141 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:10.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-03-25T16:13:10.954 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:11.105 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:11.106 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 171 k 2024-03-25T16:13:11.107 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 494 k 2024-03-25T16:13:11.107 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:13:11.336 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 748 kB/s | 171 kB 00:00 2024-03-25T16:13:11.337 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:11.337 INFO:teuthology.orchestra.run.smithi096.stdout:Total 744 kB/s | 171 kB 00:00 2024-03-25T16:13:11.337 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:13:11.345 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:13:11.345 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:13:11.401 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:13:11.402 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:13:11.556 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:13:11.660 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:12.128 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:12.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:12.664 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:13:13.235 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:13.388 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:13.388 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:13.388 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:13:13.388 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-03-25T16:13:13.389 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:13.390 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 91 M 2024-03-25T16:13:13.390 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 498 M 2024-03-25T16:13:13.390 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:13:14.839 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 15 MB/s | 22 MB 00:01 2024-03-25T16:13:17.621 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 4.2 MB/s | 68 MB 00:16 2024-03-25T16:13:17.623 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:17.623 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.6 MB/s | 91 MB 00:16 2024-03-25T16:13:17.624 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:13:17.628 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:13:17.628 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:13:18.216 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:13:18.217 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:13:19.663 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 11 MB/s | 68 MB 00:06 2024-03-25T16:13:19.665 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:19.665 INFO:teuthology.orchestra.run.smithi096.stdout:Total 14 MB/s | 91 MB 00:06 2024-03-25T16:13:19.666 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:13:19.670 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:13:19.671 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:13:20.259 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:13:20.259 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:13:21.326 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:13:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:13:28.454 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:13:28.522 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:13:30.424 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:13:30.506 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:13:32.276 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:32.276 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:32.276 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:13:32.276 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:32.277 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:32.277 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:32.277 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:13:32.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:13:33.070 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:46 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:13:33.221 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 416 M 2024-03-25T16:13:33.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.7 G 2024-03-25T16:13:33.223 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:13:33.890 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:13:33.890 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:13:34.150 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:34.151 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:13:34.370 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:13:35.017 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:37 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:13:35.173 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 416 M 2024-03-25T16:13:35.174 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.7 G 2024-03-25T16:13:35.175 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:14:51.530 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 416 MB 01:16 2024-03-25T16:14:51.530 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:51.530 INFO:teuthology.orchestra.run.smithi096.stdout:Total 5.4 MB/s | 416 MB 01:16 2024-03-25T16:14:51.531 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:14:51.533 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:14:51.534 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:14:53.993 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:14:53.993 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:15:00.642 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 4.8 MB/s | 416 MB 01:27 2024-03-25T16:15:00.642 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:00.643 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.8 MB/s | 416 MB 01:27 2024-03-25T16:15:00.643 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:15:00.648 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:15:00.648 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:15:03.115 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:15:03.115 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:15:36.200 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:15:36.304 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:44.303 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:44.545 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:15:44.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:15:45.246 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:15:45.344 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:45.352 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:15:45.506 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:45.507 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================== 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 1.4 M 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 6.4 M 2024-03-25T16:15:45.508 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:15:49.040 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 414 kB/s | 1.4 MB 00:03 2024-03-25T16:15:49.041 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:49.041 INFO:teuthology.orchestra.run.smithi096.stdout:Total 413 kB/s | 1.4 MB 00:03 2024-03-25T16:15:49.042 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:15:49.046 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:15:49.046 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:15:49.089 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:15:49.090 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:15:49.145 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:49.340 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:49.395 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:15:49.438 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:49.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:15:49.830 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:50.101 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:50.102 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:50.102 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:15:50.102 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:50.102 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:50.102 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:15:50.255 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:15:50.294 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:15:50.403 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:15:50.404 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:50.405 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.4 M 2024-03-25T16:15:50.405 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 6.4 M 2024-03-25T16:15:50.405 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:15:50.853 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:15:51.001 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:15:51.002 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:15:51.003 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:15:51.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:15:51.003 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 176 M 2024-03-25T16:15:51.003 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 753 M 2024-03-25T16:15:51.003 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:15:51.558 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 1.2 MB/s | 1.4 MB 00:01 2024-03-25T16:15:51.558 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:51.559 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.2 MB/s | 1.4 MB 00:01 2024-03-25T16:15:51.559 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:15:51.562 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:15:51.562 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:15:51.604 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:15:51.604 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:15:51.861 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:15:51.964 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:52.368 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:52.653 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:15:52.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:52.653 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:15:52.653 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:52.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:52.654 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:15:52.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:15:53.428 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:15:53.576 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:15:53.577 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:15:53.578 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 176 M 2024-03-25T16:15:53.578 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 753 M 2024-03-25T16:15:53.578 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:16:42.293 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 3.4 MB/s | 176 MB 00:51 2024-03-25T16:16:42.294 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:42.294 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.4 MB/s | 176 MB 00:51 2024-03-25T16:16:42.294 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:16:42.297 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:16:42.297 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:16:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:16:43.358 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:17:01.569 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:17:01.664 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:17:06.309 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:17:06.573 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:17:06.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:17:07.352 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:06:09 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:17:07.501 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:17:07.503 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:17:07.503 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:17:07.503 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:17:07.503 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:17:07.503 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:17:07.504 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 672 M 2024-03-25T16:17:07.505 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 3.1 G 2024-03-25T16:17:07.505 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:19:07.744 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 926 kB/s | 176 MB 03:14 2024-03-25T16:19:07.745 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:07.745 INFO:teuthology.orchestra.run.smithi203.stdout:Total 926 kB/s | 176 MB 03:14 2024-03-25T16:19:07.745 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:19:07.748 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:19:07.748 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:19:08.806 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:19:08.806 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:19:26.810 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:19:26.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:19:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:19:31.883 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:19:31.883 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:19:31.884 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:19:31.884 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:31.884 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:19:31.884 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:19:32.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:19:32.672 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:08:45 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:19:32.822 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:19:32.822 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:19:32.822 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:19:32.825 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:19:32.825 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:19:32.825 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:19:32.825 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:19:32.825 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 672 M 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 3.1 G 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:38:19.689 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 541 kB/s | 672 MB 21:12 2024-03-25T16:38:19.710 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:19.710 INFO:teuthology.orchestra.run.smithi096.stdout:Total 541 kB/s | 672 MB 21:12 2024-03-25T16:38:19.710 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:38:19.711 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:38:19.711 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:38:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:38:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:39:34.120 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:39:34.300 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:39:43.464 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:39:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:39:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:39:43.745 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:39:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:39:43.745 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:39:44.036 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:39:44.609 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:28:46 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:39:44.712 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:44.768 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:39:44.768 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-03-25T16:39:44.769 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:39:44.830 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:39:45.419 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:28:47 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:39:45.574 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:39:45.575 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:39:45.575 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:39:45.575 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:39:45.575 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:39:45.575 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 20 M 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 103 M 2024-03-25T16:39:45.576 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:40:34.128 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 415 kB/s | 20 MB 00:48 2024-03-25T16:40:34.128 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:34.129 INFO:teuthology.orchestra.run.smithi096.stdout:Total 415 kB/s | 20 MB 00:48 2024-03-25T16:40:34.129 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:40:34.132 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:40:34.133 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:40:34.281 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:40:34.281 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:40:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:40:36.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:40:37.627 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:40:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:40:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:40:37.892 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:40:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:40:37.893 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:40:38.062 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:40:38.620 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:29:40 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:40:38.767 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:40:38.767 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:40:38.767 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:40:38.767 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:40:38.768 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 48 M 2024-03-25T16:40:38.769 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 173 M 2024-03-25T16:40:38.769 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:42:36.157 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 416 kB/s | 48 MB 01:57 2024-03-25T16:42:36.158 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:42:36.158 INFO:teuthology.orchestra.run.smithi096.stdout:Total 416 kB/s | 48 MB 01:57 2024-03-25T16:42:36.158 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:42:36.161 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:42:36.161 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:42:36.471 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:42:36.472 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:42:41.259 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:42:41.354 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:42:42.675 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:42:42.954 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:42:43.145 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:42:43.708 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:31:45 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:42:43.856 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 99 M 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 449 M 2024-03-25T16:42:43.857 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:49:09.007 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 388 kB/s | 672 MB 29:36 2024-03-25T16:49:09.007 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:09.007 INFO:teuthology.orchestra.run.smithi203.stdout:Total 388 kB/s | 672 MB 29:36 2024-03-25T16:49:09.007 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:49:09.012 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:49:09.012 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:49:12.990 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:49:12.990 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:50:23.928 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:50:24.291 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 219 kB/s | 99 MB 07:40 2024-03-25T16:50:24.291 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:50:24.292 INFO:teuthology.orchestra.run.smithi096.stdout:Total 219 kB/s | 99 MB 07:40 2024-03-25T16:50:24.292 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T16:50:24.296 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T16:50:24.296 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T16:50:24.534 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:50:24.902 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T16:50:24.903 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T16:50:32.863 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:50:33.249 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:50:33.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:50:34.134 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:39:47 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:50:34.230 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:50:34.283 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:50:34.284 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T16:50:34.284 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:50:34.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:50:34.923 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:39:47 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:50:35.071 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:50:35.071 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 20 M 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 103 M 2024-03-25T16:50:35.072 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:50:35.132 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T16:50:35.238 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:50:38.032 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:50:38.290 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T16:50:38.504 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T16:50:39.063 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:39:41 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T16:50:39.210 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T16:50:39.211 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T16:50:39.212 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T16:50:39.212 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T16:50:39.212 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 287 M 2024-03-25T16:50:39.212 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.2 G 2024-03-25T16:50:39.212 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T16:51:54.027 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 255 kB/s | 20 MB 01:18 2024-03-25T16:51:54.028 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:51:54.028 INFO:teuthology.orchestra.run.smithi203.stdout:Total 255 kB/s | 20 MB 01:18 2024-03-25T16:51:54.028 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:51:54.031 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:51:54.032 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:51:54.180 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:51:54.180 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:51:56.401 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:51:56.566 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:51:57.825 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:51:58.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:51:58.582 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:41:11 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:51:58.730 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:51:58.731 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:51:58.731 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:51:58.731 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:51:58.731 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 173 M 2024-03-25T16:51:58.732 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T16:54:11.621 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 48 MB 02:12 2024-03-25T16:54:11.621 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:54:11.621 INFO:teuthology.orchestra.run.smithi203.stdout:Total 368 kB/s | 48 MB 02:12 2024-03-25T16:54:11.621 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T16:54:11.624 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T16:54:11.624 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T16:54:11.937 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T16:54:11.938 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T16:54:16.695 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T16:54:16.801 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:54:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:54:18.496 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:54:18.496 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:54:18.496 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T16:54:18.496 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:18.496 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:54:18.497 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T16:54:18.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:54:19.262 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:43:32 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T16:54:19.410 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T16:54:19.411 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:54:19.411 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 99 M 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 449 M 2024-03-25T16:54:19.412 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:00:46.211 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 261 kB/s | 99 MB 06:26 2024-03-25T17:00:46.211 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:00:46.211 INFO:teuthology.orchestra.run.smithi203.stdout:Total 261 kB/s | 99 MB 06:26 2024-03-25T17:00:46.212 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:00:46.214 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:00:46.215 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:00:46.822 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:00:46.823 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:00:57.130 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:00:57.240 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:01:00.310 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:01:00.563 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:01:00.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:01:01.340 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:50:14 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:01:01.492 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:01:01.492 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 287 M 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.2 G 2024-03-25T17:01:01.493 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:05:57.693 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 320 kB/s | 287 MB 15:18 2024-03-25T17:05:57.693 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:05:57.694 INFO:teuthology.orchestra.run.smithi096.stdout:Total 320 kB/s | 287 MB 15:18 2024-03-25T17:05:57.694 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:05:57.696 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:05:57.697 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:05:59.400 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:05:59.400 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:06:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:06:29.195 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:32.576 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:32.857 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:06:33.090 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:06:33.651 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:55:35 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:06:33.799 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:06:33.800 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 5.9 M 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 28 M 2024-03-25T17:06:33.801 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:06:51.978 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 335 kB/s | 5.9 MB 00:18 2024-03-25T17:06:51.979 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:51.979 INFO:teuthology.orchestra.run.smithi096.stdout:Total 335 kB/s | 5.9 MB 00:18 2024-03-25T17:06:51.980 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:06:51.983 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:06:51.984 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:06:52.054 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:06:52.055 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:06:52.793 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:06:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:06:53.511 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:53.773 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:06:53.966 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:06:54.524 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:55:56 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:06:54.673 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 109 k 2024-03-25T17:06:54.674 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 440 k 2024-03-25T17:06:54.675 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:06:55.515 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 129 kB/s | 109 kB 00:00 2024-03-25T17:06:55.515 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:55.516 INFO:teuthology.orchestra.run.smithi096.stdout:Total 129 kB/s | 109 kB 00:00 2024-03-25T17:06:55.516 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:06:55.520 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:06:55.520 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:06:55.556 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:06:55.556 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:06:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:06:55.796 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:06:56.204 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:56.462 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:06:56.670 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:06:57.229 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:55:59 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:06:57.376 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 5.4 M 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 25 M 2024-03-25T17:06:57.377 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:07:10.091 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 431 kB/s | 5.4 MB 00:12 2024-03-25T17:07:10.091 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:07:10.092 INFO:teuthology.orchestra.run.smithi096.stdout:Total 431 kB/s | 5.4 MB 00:12 2024-03-25T17:07:10.092 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:07:10.096 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:07:10.096 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:07:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:07:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:07:10.822 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:07:10.931 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:07:11.427 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:07:11.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:07:11.695 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:07:11.695 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:07:11.695 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:11.695 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:07:11.695 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:07:11.895 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2-debuginfo 2024-03-25T17:07:12.477 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:56:14 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:07:12.635 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:07:12.636 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:07:12.637 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:07:12.637 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:07:12.637 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 44 M 2024-03-25T17:07:12.637 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 253 M 2024-03-25T17:07:12.637 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:09:59.554 INFO:teuthology.orchestra.run.smithi096.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 272 kB/s | 44 MB 02:46 2024-03-25T17:09:59.555 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:59.555 INFO:teuthology.orchestra.run.smithi096.stdout:Total 272 kB/s | 44 MB 02:46 2024-03-25T17:09:59.555 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:09:59.557 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:09:59.557 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:09:59.856 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:09:59.856 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:10:04.894 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:10:04.991 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:10:07.011 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:10:07.263 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:10:07.456 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:10:08.017 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:59:10 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:10:08.166 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:10:08.167 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:10:08.168 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 52 M 2024-03-25T17:10:08.168 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 308 M 2024-03-25T17:10:08.168 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:12:36.868 INFO:teuthology.orchestra.run.smithi096.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 357 kB/s | 52 MB 02:28 2024-03-25T17:12:36.869 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:12:36.869 INFO:teuthology.orchestra.run.smithi096.stdout:Total 357 kB/s | 52 MB 02:28 2024-03-25T17:12:36.869 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:12:36.872 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:12:36.872 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:12:37.213 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:12:37.214 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:12:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:12:43.339 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:12:45.498 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:12:45.785 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:12:45.786 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:45.786 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:12:45.786 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:45.786 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:45.786 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:12:46.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:12:46.579 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:01:48 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:12:46.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 615 k 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.1 M 2024-03-25T17:12:46.729 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:12:49.152 INFO:teuthology.orchestra.run.smithi096.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 254 kB/s | 615 kB 00:02 2024-03-25T17:12:49.153 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:12:49.153 INFO:teuthology.orchestra.run.smithi096.stdout:Total 254 kB/s | 615 kB 00:02 2024-03-25T17:12:49.153 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:12:49.157 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:12:49.157 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:12:49.199 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:12:49.200 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:12:49.362 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:12:49.477 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:12:49.875 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:12:50.133 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:12:50.134 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:50.134 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:12:50.134 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:50.134 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:50.134 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:12:50.331 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:12:50.890 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:01:52 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:12:51.038 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:51.039 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 771 k 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.7 M 2024-03-25T17:12:51.040 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:12:54.422 INFO:teuthology.orchestra.run.smithi096.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 228 kB/s | 771 kB 00:03 2024-03-25T17:12:54.422 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:12:54.423 INFO:teuthology.orchestra.run.smithi096.stdout:Total 228 kB/s | 771 kB 00:03 2024-03-25T17:12:54.423 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:12:54.427 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:12:54.427 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:12:54.468 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:12:54.468 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:12:54.654 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:12:54.754 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:12:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:12:55.385 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:12:55.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:55.385 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:12:55.386 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:55.386 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:55.386 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:12:55.636 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:12:56.194 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:01:58 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:12:56.344 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:12:56.345 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:56.345 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 1.1 M 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 4.3 M 2024-03-25T17:12:56.346 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:13:03.382 INFO:teuthology.orchestra.run.smithi096.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 164 kB/s | 1.1 MB 00:07 2024-03-25T17:13:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:03.383 INFO:teuthology.orchestra.run.smithi096.stdout:Total 164 kB/s | 1.1 MB 00:07 2024-03-25T17:13:03.384 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:13:03.388 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:13:03.388 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:13:03.431 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:13:03.431 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:13:03.672 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:13:03.764 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:04.175 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:04.440 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:04.441 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:04.441 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:13:04.441 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:04.441 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:04.441 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:13:04.673 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:13:05.234 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:02:07 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:13:05.383 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:13:05.384 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:05.384 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:13:05.384 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 337 k 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.0 M 2024-03-25T17:13:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:13:06.828 INFO:teuthology.orchestra.run.smithi096.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 234 kB/s | 337 kB 00:01 2024-03-25T17:13:06.829 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:06.829 INFO:teuthology.orchestra.run.smithi096.stdout:Total 234 kB/s | 337 kB 00:01 2024-03-25T17:13:06.830 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:13:06.833 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:13:06.833 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:13:06.872 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:13:06.873 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:13:07.016 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:13:07.119 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:07.467 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:07.731 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:13:07.937 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:13:08.496 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:02:10 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:13:08.647 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:13:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 509 k 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.5 M 2024-03-25T17:13:08.649 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:13:11.714 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 166 kB/s | 509 kB 00:03 2024-03-25T17:13:11.715 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:11.716 INFO:teuthology.orchestra.run.smithi096.stdout:Total 166 kB/s | 509 kB 00:03 2024-03-25T17:13:11.716 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:13:11.720 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:13:11.720 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:13:11.760 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:13:11.760 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:13:11.922 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:13:12.037 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:13:12.410 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:12.671 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:13:12.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:13:13.430 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:02:15 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:13:13.580 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 52 M 2024-03-25T17:13:13.581 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 298 M 2024-03-25T17:13:13.582 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:16:21.472 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 281 kB/s | 52 MB 03:07 2024-03-25T17:16:21.473 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:21.473 INFO:teuthology.orchestra.run.smithi096.stdout:Total 281 kB/s | 52 MB 03:07 2024-03-25T17:16:21.473 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:16:21.476 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:16:21.476 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:16:21.809 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:16:21.809 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:16:27.739 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:16:27.994 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:30.102 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:30.364 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:16:30.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:16:31.129 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 1:05:33 ago on Mon 25 Mar 2024 04:10:58 PM UTC. 2024-03-25T17:16:31.277 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-25T17:16:31.278 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:16:31.278 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-03-25T17:16:31.278 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 1.3 M 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 5.5 M 2024-03-25T17:16:31.279 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-25T17:16:36.239 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 259 kB/s | 1.3 MB 00:04 2024-03-25T17:16:36.239 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:36.240 INFO:teuthology.orchestra.run.smithi096.stdout:Total 258 kB/s | 1.3 MB 00:04 2024-03-25T17:16:36.240 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-25T17:16:36.244 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-25T17:16:36.244 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-25T17:16:36.288 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-25T17:16:36.289 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-25T17:16:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-25T17:16:36.637 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:37.080 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:16:37.344 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-25T17:16:37.557 DEBUG:teuthology.parallel:result is None 2024-03-25T17:18:06.303 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 287 kB/s | 287 MB 17:04 2024-03-25T17:18:06.303 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:18:06.303 INFO:teuthology.orchestra.run.smithi203.stdout:Total 287 kB/s | 287 MB 17:04 2024-03-25T17:18:06.303 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:18:06.306 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:18:06.306 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:18:08.016 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:18:08.016 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:18:37.551 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:18:37.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:18:45.857 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:18:46.127 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:18:46.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:18:46.949 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:07:59 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:18:47.098 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:18:47.099 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.9 M 2024-03-25T17:18:47.100 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 28 M 2024-03-25T17:18:47.101 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:19:09.139 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 276 kB/s | 5.9 MB 00:22 2024-03-25T17:19:09.141 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:09.141 INFO:teuthology.orchestra.run.smithi203.stdout:Total 276 kB/s | 5.9 MB 00:22 2024-03-25T17:19:09.141 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:19:09.145 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:19:09.145 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:19:09.217 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:19:09.217 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:19:09.956 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:19:10.055 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:19:10.654 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:19:11.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:19:11.677 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:08:24 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:19:11.828 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:19:11.829 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:11.830 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 109 k 2024-03-25T17:19:11.830 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 440 k 2024-03-25T17:19:11.830 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:19:12.917 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 100 kB/s | 109 kB 00:01 2024-03-25T17:19:12.917 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:12.918 INFO:teuthology.orchestra.run.smithi203.stdout:Total 100 kB/s | 109 kB 00:01 2024-03-25T17:19:12.918 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:19:12.921 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:19:12.921 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:19:12.956 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:19:12.957 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:19:13.076 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:19:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:19:13.560 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:19:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:19:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:13.818 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:19:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:19:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:13.818 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:19:14.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:19:14.577 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:08:27 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:19:14.726 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:19:14.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:14.728 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.4 M 2024-03-25T17:19:14.728 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 25 M 2024-03-25T17:19:14.728 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:19:37.909 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 236 kB/s | 5.4 MB 00:23 2024-03-25T17:19:37.910 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:37.910 INFO:teuthology.orchestra.run.smithi203.stdout:Total 236 kB/s | 5.4 MB 00:23 2024-03-25T17:19:37.911 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:19:37.914 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:19:37.915 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:19:37.982 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:19:37.983 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:19:38.646 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:19:38.742 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:19:39.251 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:39.497 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:19:39.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2-debuginfo 2024-03-25T17:19:40.308 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:08:53 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:19:40.471 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 44 M 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 253 M 2024-03-25T17:19:40.472 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:22:25.257 INFO:teuthology.orchestra.run.smithi203.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 275 kB/s | 44 MB 02:44 2024-03-25T17:22:25.258 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:22:25.258 INFO:teuthology.orchestra.run.smithi203.stdout:Total 275 kB/s | 44 MB 02:44 2024-03-25T17:22:25.258 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:22:25.262 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:22:25.262 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:22:25.562 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:22:25.562 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:22:30.714 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:32.726 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:32.986 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:32.986 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:22:32.987 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:22:32.987 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:22:32.987 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:22:32.987 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:22:33.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:22:33.755 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:11:46 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:22:33.905 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:22:33.905 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:22:33.905 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 308 M 2024-03-25T17:22:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:12.763 INFO:teuthology.orchestra.run.smithi203.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 334 kB/s | 52 MB 02:38 2024-03-25T17:25:12.763 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:12.764 INFO:teuthology.orchestra.run.smithi203.stdout:Total 334 kB/s | 52 MB 02:38 2024-03-25T17:25:12.764 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:12.767 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:12.767 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:13.112 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:13.112 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:19.244 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:21.497 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:21.752 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:21.752 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:21.752 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:21.752 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:21.753 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:21.753 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:21.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:25:22.516 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:35 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:22.668 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:22.668 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:22.669 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:22.670 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 615 k 2024-03-25T17:25:22.670 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.1 M 2024-03-25T17:25:22.670 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:25.481 INFO:teuthology.orchestra.run.smithi203.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 219 kB/s | 615 kB 00:02 2024-03-25T17:25:25.481 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:25.482 INFO:teuthology.orchestra.run.smithi203.stdout:Total 219 kB/s | 615 kB 00:02 2024-03-25T17:25:25.482 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:25.486 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:25.486 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:25.527 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:25.528 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:25.693 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:25.805 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:26.181 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:26.434 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:26.435 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:26.435 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:26.435 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:26.435 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:26.435 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:26.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:25:27.210 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:40 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:27.362 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:27.363 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:27.363 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 771 k 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.7 M 2024-03-25T17:25:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:30.942 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 215 kB/s | 771 kB 00:03 2024-03-25T17:25:30.943 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:30.943 INFO:teuthology.orchestra.run.smithi203.stdout:Total 215 kB/s | 771 kB 00:03 2024-03-25T17:25:30.943 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:30.947 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:30.947 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:30.990 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:30.990 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:31.200 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:31.291 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:25:31.667 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:31.933 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:32.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:25:32.723 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:45 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:32.872 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:32.873 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.1 M 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 4.3 M 2024-03-25T17:25:32.874 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:35.603 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 422 kB/s | 1.1 MB 00:02 2024-03-25T17:25:35.603 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:35.604 INFO:teuthology.orchestra.run.smithi203.stdout:Total 421 kB/s | 1.1 MB 00:02 2024-03-25T17:25:35.604 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:35.608 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:35.608 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:35.652 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:35.652 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:35.870 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:35.977 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:36.363 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:36.624 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:36.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:25:37.400 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:50 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:37.551 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:37.552 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:37.552 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:37.552 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:37.552 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:37.552 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 337 k 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.0 M 2024-03-25T17:25:37.553 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:38.819 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 267 kB/s | 337 kB 00:01 2024-03-25T17:25:38.819 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:38.819 INFO:teuthology.orchestra.run.smithi203.stdout:Total 267 kB/s | 337 kB 00:01 2024-03-25T17:25:38.819 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:38.820 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:38.820 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:38.859 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:38.859 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:38.997 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:39.104 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:39.475 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:39.739 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:39.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:25:40.522 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:53 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:40.672 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:25:40.673 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 509 k 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.5 M 2024-03-25T17:25:40.674 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:25:42.267 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 320 kB/s | 509 kB 00:01 2024-03-25T17:25:42.268 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:42.268 INFO:teuthology.orchestra.run.smithi203.stdout:Total 319 kB/s | 509 kB 00:01 2024-03-25T17:25:42.268 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:25:42.272 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:25:42.272 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:25:42.312 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:25:42.313 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:25:42.476 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:25:42.591 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:25:42.987 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:25:43.246 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:25:43.246 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:43.246 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:25:43.247 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:43.247 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:43.247 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:25:43.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:25:44.032 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:14:57 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:25:44.182 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:25:44.183 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:25:44.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:25:44.185 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-03-25T17:25:44.185 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 298 M 2024-03-25T17:25:44.185 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:27:44.338 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 439 kB/s | 52 MB 02:00 2024-03-25T17:27:44.339 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:27:44.339 INFO:teuthology.orchestra.run.smithi203.stdout:Total 439 kB/s | 52 MB 02:00 2024-03-25T17:27:44.339 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:27:44.342 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:27:44.342 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:27:44.677 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:27:44.678 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:27:50.604 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:27:50.885 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:27:53.113 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:53.368 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:27:53.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:27:54.158 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:17:07 ago on Mon 25 Mar 2024 04:10:47 PM UTC. 2024-03-25T17:27:54.307 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:54.308 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.3 M 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 5.5 M 2024-03-25T17:27:54.309 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:27:58.113 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 337 kB/s | 1.3 MB 00:03 2024-03-25T17:27:58.114 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:27:58.115 INFO:teuthology.orchestra.run.smithi203.stdout:Total 337 kB/s | 1.3 MB 00:03 2024-03-25T17:27:58.115 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:27:58.119 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:27:58.119 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:27:58.163 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:27:58.164 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:27:58.423 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:27:58.530 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:27:58.987 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:27:59.243 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:27:59.243 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:59.244 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:27:59.244 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:27:59.244 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:27:59.244 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:27:59.441 DEBUG:teuthology.parallel:result is None 2024-03-25T17:27:59.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:27:59.660 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:27:59.685 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:27:59.685 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:27:59.685 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:27:59.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:27:59.904 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:27:59.924 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:27:59.925 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:27:59.925 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:27:59.926 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T17:27:59.927 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:27:59.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:27:59.959 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:27:59.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:27:59.992 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T17:27:59.992 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:27:59.992 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:28:00.029 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:28:00.098 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:00.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:28:00.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:28:00.191 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T17:28:00.191 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:00.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:28:00.220 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:28:00.289 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:00.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:28:00.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:28:00.382 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T17:28:00.382 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:00.382 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:28:00.410 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:28:00.478 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:00.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:28:00.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:28:00.572 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T17:28:00.752 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T17:28:00.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-03-25T17:28:00.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-03-25T17:28:00.781 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T17:28:00.781 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:28:00.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:28:00.848 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T17:28:00.848 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:28:00.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:28:00.947 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T17:28:00.947 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T17:28:00.947 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621240', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGhOWy56aoh5UtNOg3LmvNGF8b0GwQ4bUk7rw2EVB84w1QfE8UxSmJcQHHMGLXeWpLlkJ1tM67+kxnDCBO/sQI=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOvNpSlpcxMfV8QgjnjUklGa74zPD66ymf4Wxg95gUb/NB/Bhwumx03H/2i9zqOSV3uA5AUxbS8D+tIIdcOwm8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T17:28:00.947 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:28:00.956 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:28:00.970 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:28:01.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:28:01.037 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:01.037 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:28:01.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:28:01.091 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-25 16:13:09.243776078 +0000 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-25 16:10:03.443825335 +0000 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-25 16:10:03.443825335 +0000 2024-03-25T17:28:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-25 16:10:03.443825335 +0000 2024-03-25T17:28:01.145 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:28:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-25T17:28:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-25T17:28:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000206194 s, 2.5 MB/s 2024-03-25T17:28:01.210 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:28:01.266 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-25 16:13:09.243776078 +0000 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-25 16:10:03.793817116 +0000 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-25 16:10:03.793817116 +0000 2024-03-25T17:28:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-25 16:10:03.793817116 +0000 2024-03-25T17:28:01.321 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:28:01.385 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-25T17:28:01.385 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-25T17:28:01.385 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000189758 s, 2.7 MB/s 2024-03-25T17:28:01.386 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:28:01.441 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-25 16:13:09.244776055 +0000 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-25 16:10:04.145808850 +0000 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-25 16:10:04.145808850 +0000 2024-03-25T17:28:01.496 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-25 16:10:04.145808850 +0000 2024-03-25T17:28:01.497 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:28:01.561 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-25T17:28:01.562 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-25T17:28:01.562 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000180968 s, 2.8 MB/s 2024-03-25T17:28:01.563 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:28:01.618 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-25 16:13:09.244776055 +0000 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-25 16:10:04.493800678 +0000 2024-03-25T17:28:01.672 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-25 16:10:04.493800678 +0000 2024-03-25T17:28:01.673 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-25 16:10:04.493800678 +0000 2024-03-25T17:28:01.673 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:28:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-25T17:28:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-25T17:28:01.737 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000184798 s, 2.8 MB/s 2024-03-25T17:28:01.739 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:28:01.794 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:01.794 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:01.794 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:28:01.810 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:28:01.810 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-25T17:28:01.866 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:28:01.866 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:01.866 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T17:28:01.866 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 16:12:56.761252408 +0000 2024-03-25T17:28:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 16:10:03.461798617 +0000 2024-03-25T17:28:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 16:10:03.461798617 +0000 2024-03-25T17:28:01.867 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 16:10:03.461798617 +0000 2024-03-25T17:28:01.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:28:01.933 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T17:28:01.933 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T17:28:01.933 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000210691 s, 2.4 MB/s 2024-03-25T17:28:01.934 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:28:01.991 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 16:12:56.761252408 +0000 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 16:10:03.817789629 +0000 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 16:10:03.817789629 +0000 2024-03-25T17:28:02.046 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 16:10:03.817789629 +0000 2024-03-25T17:28:02.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:28:02.112 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T17:28:02.112 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T17:28:02.112 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000290903 s, 1.8 MB/s 2024-03-25T17:28:02.113 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:28:02.168 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 16:12:56.762252384 +0000 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 16:10:04.178780513 +0000 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 16:10:04.178780513 +0000 2024-03-25T17:28:02.224 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 16:10:04.178780513 +0000 2024-03-25T17:28:02.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:28:02.289 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T17:28:02.289 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T17:28:02.289 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000208709 s, 2.5 MB/s 2024-03-25T17:28:02.290 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:28:02.345 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-25T17:28:02.399 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 16:12:56.762252384 +0000 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 16:10:04.524771777 +0000 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 16:10:04.524771777 +0000 2024-03-25T17:28:02.400 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 16:10:04.524771777 +0000 2024-03-25T17:28:02.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:28:02.465 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T17:28:02.465 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T17:28:02.465 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000199536 s, 2.6 MB/s 2024-03-25T17:28:02.466 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:28:02.522 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:02.522 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:28:02.522 INFO:tasks.ceph:Generating config... 2024-03-25T17:28:02.525 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T17:28:02.525 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T17:28:02.525 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T17:28:02.525 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T17:28:02.525 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T17:28:02.526 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T17:28:02.526 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T17:28:02.526 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T17:28:02.526 INFO:tasks.ceph:[global] ms type = async 2024-03-25T17:28:02.526 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T17:28:02.526 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T17:28:02.527 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T17:28:02.527 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T17:28:02.527 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T17:28:02.528 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T17:28:02.529 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T17:28:02.529 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T17:28:02.529 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T17:28:02.529 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:02.595 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T17:28:02.597 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T17:28:02.642 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:02.668 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.96:6789]'), ('mon.c', '[v1:172.21.15.96:6790]'), ('mon.b', '[v1:172.21.15.203:6789]')] 2024-03-25T17:28:02.668 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.96:6789],[v1:172.21.15.96:6790],[v1:172.21.15.203:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T17:28:02.669 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:02.669 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:28:02.723 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.96:6789]' --addv c '[v1:172.21.15.96:6790]' --addv b '[v1:172.21.15.203:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:02.807 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:02.807 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:28:02.807 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = quincy 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 17 (quincy) 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 3 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:28:02.808 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T17:28:02.809 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:28:02.864 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7a945e62-7593-49b6-a8a4-685c9e0f96b4... 2024-03-25T17:28:02.864 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:28:02.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.954 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:28:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-03-25T17:28:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "[v1:172.21.15.96:6789],[v1:172.21.15.96:6790],[v1:172.21.15.203:6789]" 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: mon election default strategy = 3 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: ms bind msgr2 = False 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-03-25T17:28:02.957 INFO:teuthology.orchestra.run.smithi096.stdout: ms type = async 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd heartbeat grace = 80 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-03-25T17:28:02.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-03-25T17:28:02.959 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bdev async discard = True 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bdev enable discard = True 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore allocator = stupid 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore zero block detection = True 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1 2024-03-25T17:28:02.960 INFO:teuthology.orchestra.run.smithi096.stdout: debug monc = 20 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 10 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd class default list = * 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd class load list = * 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd client watch timeout = 120 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify cached snaps = True 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify missing on start = True 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd fast shutdown = False 2024-03-25T17:28:02.961 INFO:teuthology.orchestra.run.smithi096.stdout: osd max markdown count = 1000 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock override recovery settings = True 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: osd op queue = debug_random 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: osd op queue cut off = debug_random 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: osd shutdown pgref assert = True 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-03-25T17:28:02.962 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:28:02.963 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon min osdmap epochs = 50 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd crush smoke test = False 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon osdmap full prune interval = 2 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon osdmap full prune min = 15 2024-03-25T17:28:02.964 INFO:teuthology.orchestra.run.smithi096.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: mon scrub interval = 300 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: paxos service trim min = 10 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: debug monc = 20 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: debug objecter = 20 2024-03-25T17:28:02.965 INFO:teuthology.orchestra.run.smithi096.stdout: debug rados = 20 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:28:02.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.968 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.96:6789],[v1:172.21.15.96:6790],[v1:172.21.15.203:6789]" 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 3 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 5000 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: osd heartbeat grace = 80 2024-03-25T17:28:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-03-25T17:28:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.971 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore allocator = stupid 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 1 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 1 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:02.972 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd class default list = * 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd class load list = * 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd client watch timeout = 120 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd fast shutdown = False 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd max markdown count = 1000 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-03-25T17:28:02.973 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T17:28:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:28:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd crush smoke test = False 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on pool no app = False 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-03-25T17:28:02.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: debug objecter = 20 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout: debug rados = 20 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-03-25T17:28:02.977 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-03-25T17:28:02.979 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T17:28:02.979 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T17:28:03.033 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T17:28:03.033 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:03.033 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T17:28:03.048 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:03.048 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T17:28:03.103 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:28:03.103 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:03.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:28:03.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:03.182 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:03.182 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:03.237 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:28:03.237 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:03.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:28:03.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:03.277 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:03.277 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:03.332 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T17:28:03.333 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T17:28:03.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:28:03.387 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:28:03.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:28:03.449 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:28:03.452 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T17:28:03.452 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T17:28:03.452 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T17:28:03.493 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T17:28:03.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T17:28:03.551 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T17:28:03.566 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T17:28:03.566 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:28:03.566 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:03.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:03.593 INFO:tasks.ceph:role: osd.0 2024-03-25T17:28:03.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:28:03.593 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:28:03.667 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:03.667 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:03.667 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:03.668 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:04.185 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-03-25T17:28:04.187 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-03-25T17:28:04.187 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:04.307 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:04.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:04.411 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:04.412 INFO:tasks.ceph:role: osd.1 2024-03-25T17:28:04.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:28:04.412 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:04.487 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:04.488 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:04.488 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:04.957 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-03-25T17:28:04.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-03-25T17:28:04.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:05.050 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:05.081 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:05.149 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:05.149 INFO:tasks.ceph:role: osd.2 2024-03-25T17:28:05.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:28:05.149 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:05.224 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:05.715 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-03-25T17:28:05.718 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-03-25T17:28:05.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:05.840 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:05.874 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:05.942 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:05.942 INFO:tasks.ceph:role: osd.3 2024-03-25T17:28:05.942 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:28:05.942 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:28:06.016 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:06.017 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:06.536 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-03-25T17:28:06.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-03-25T17:28:06.539 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:06.676 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:06.710 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:06.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.813+0000 7f6d9baf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T17:28:06.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.813+0000 7f6d9baf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T17:28:06.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.813+0000 7f6d9baf9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:06.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.813+0000 7f6d9baf9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:06.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.813+0000 7f6d9baf9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:06.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:06.814+0000 7f6d9baf9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T17:28:09.661 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:09.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.783+0000 7fd30b210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.784+0000 7fd30b210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.784+0000 7fd30b210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.784+0000 7fd30b210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.784+0000 7fd30b210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:09.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:09.784+0000 7fd30b210640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T17:28:12.628 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:12.660 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:12.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.749+0000 7f82c0024640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T17:28:12.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.749+0000 7f82c0024640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T17:28:12.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.749+0000 7f82c0024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:12.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.749+0000 7f82c0024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:12.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.749+0000 7f82c0024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:12.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:12.750+0000 7f82c0024640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T17:28:15.596 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:15.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.720+0000 7fe736211640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.720+0000 7fe736211640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.720+0000 7fe736211640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.720+0000 7fe736211640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.720+0000 7fe736211640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:15.721+0000 7fe736211640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T17:28:18.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:18.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:18.628 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:18.628 INFO:tasks.ceph:role: osd.4 2024-03-25T17:28:18.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:28:18.628 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:28:18.707 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:18.708 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:19.222 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T17:28:19.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T17:28:19.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:19.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:19.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:19.489 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:19.490 INFO:tasks.ceph:role: osd.5 2024-03-25T17:28:19.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:28:19.490 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:28:19.565 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:19.566 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:20.013 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T17:28:20.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T17:28:20.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:20.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:20.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:20.222 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:20.222 INFO:tasks.ceph:role: osd.6 2024-03-25T17:28:20.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:28:20.222 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:28:20.297 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:20.297 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:20.297 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:20.297 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:20.298 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:20.784 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T17:28:20.786 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T17:28:20.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:20.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:20.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:21.038 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:21.038 INFO:tasks.ceph:role: osd.7 2024-03-25T17:28:21.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:28:21.038 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:21.114 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:21.115 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:21.115 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:21.115 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:21.604 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T17:28:21.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T17:28:21.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:21.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:21.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:21.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.896+0000 7f9da041c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T17:28:21.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.896+0000 7f9da041c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T17:28:21.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.896+0000 7f9da041c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:21.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.896+0000 7f9da041c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:21.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.896+0000 7f9da041c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:21.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:21.897+0000 7f9da041c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T17:28:24.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:24.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:24.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.863+0000 7f56a961b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T17:28:24.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.864+0000 7f56a961b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T17:28:24.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.864+0000 7f56a961b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:24.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.864+0000 7f56a961b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:24.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.864+0000 7f56a961b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:24.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:24.864+0000 7f56a961b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T17:28:27.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:27.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:27.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.827+0000 7fd220b9e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T17:28:27.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.827+0000 7fd220b9e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T17:28:27.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.827+0000 7fd220b9e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:27.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.827+0000 7fd220b9e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:27.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.827+0000 7fd220b9e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:27.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:27.828+0000 7fd220b9e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T17:28:30.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:30.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.793+0000 7f8a4c20c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T17:28:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.793+0000 7f8a4c20c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T17:28:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.793+0000 7f8a4c20c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.793+0000 7f8a4c20c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.793+0000 7f8a4c20c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:28:30.794+0000 7f8a4c20c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T17:28:33.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:33.660 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T17:28:33.660 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:33.660 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T17:28:33.692 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:33.693 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T17:28:33.763 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:33.763 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T17:28:33.832 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:33.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T17:28:33.901 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:33.901 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T17:28:33.970 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:33.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T17:28:34.002 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:34.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T17:28:34.072 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:34.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T17:28:34.141 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:34.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T17:28:34.210 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:34.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T17:28:34.279 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:28:34.279 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T17:28:34.296 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:28:34.296 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T17:28:34.335 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T17:28:34.335 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:28:34.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:28:34.378 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.y] 2024-03-25T17:28:34.378 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAjtAFmTQUeFxAAp0Ib6mh33FtCaxZwdGds9Q== 2024-03-25T17:28:34.378 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-03-25T17:28:34.378 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAmtAFmjeOPMBAAdaorfWRtifrKPMSyi1bmGw== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAptAFmBXPNLhAAtt5GqQEHnL1+oom9Yd1IBQ== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAstAFmK/6/LBAA717Vs6jhyS95PzYHxPLKbg== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAvtAFmHGMBKxAAU0UAs7vR34m89hKBYkIHmA== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAjtAFmqCnUGhAAhdB5ItsSFi9mRkMDVI7fTQ== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQA1tAFm8TpxNRAARb1WIXPR5B6fl3oKrF6kXA== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQA4tAFmT3eFMxAA8HYJz/fGGDhGJ4iNHE1e/A== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.6] 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQA7tAFmTndWMRAAO3sLlHLgSpSrCGCp9WXoUg== 2024-03-25T17:28:34.379 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.7] 2024-03-25T17:28:34.380 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQA+tAFmicRPLxAAbc6gfpYt85bwoWGGDHgnJA== 2024-03-25T17:28:34.380 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-03-25T17:28:34.380 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAjtAFmf0Z5HRAAvo4VWrQ7bjYQGkkWkdfjUg== 2024-03-25T17:28:34.380 INFO:teuthology.orchestra.run.smithi096.stdout:[client.1] 2024-03-25T17:28:34.380 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAjtAFm2lfiIBAABokZ5kxH5JfkbfpgxY0dfA== 2024-03-25T17:28:34.401 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2024-03-25T17:28:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAjtAFmTQUeFxAAp0Ib6mh33FtCaxZwdGds9Q== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmtAFmjeOPMBAAdaorfWRtifrKPMSyi1bmGw== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAptAFmBXPNLhAAtt5GqQEHnL1+oom9Yd1IBQ== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAstAFmK/6/LBAA717Vs6jhyS95PzYHxPLKbg== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAvtAFmHGMBKxAAU0UAs7vR34m89hKBYkIHmA== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAjtAFmqCnUGhAAhdB5ItsSFi9mRkMDVI7fTQ== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA1tAFm8TpxNRAARb1WIXPR5B6fl3oKrF6kXA== 2024-03-25T17:28:34.402 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA4tAFmT3eFMxAA8HYJz/fGGDhGJ4iNHE1e/A== 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA7tAFmTndWMRAAO3sLlHLgSpSrCGCp9WXoUg== 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA+tAFmicRPLxAAbc6gfpYt85bwoWGGDHgnJA== 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAjtAFmf0Z5HRAAvo4VWrQ7bjYQGkkWkdfjUg== 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-03-25T17:28:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAjtAFm2lfiIBAABokZ5kxH5JfkbfpgxY0dfA== 2024-03-25T17:28:34.404 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:34.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:34.497 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.604 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.658 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.711 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:34.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:34.765 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.819 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.872 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.925 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:34.979 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:34.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:35.036 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:35.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:35.093 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T17:28:35.094 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T17:28:35.120 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:35.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T17:28:35.242 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T17:28:35.309 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:35.406 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T17:28:35.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T17:28:35.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:35.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T17:28:35.582 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:35.585 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:35.637 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T17:28:35.638 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:35.638 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T17:28:35.638 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T17:28:35.640 INFO:tasks.ceph.mon.a:Started 2024-03-25T17:28:35.640 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:35.640 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T17:28:35.640 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T17:28:35.642 INFO:tasks.ceph.mon.c:Started 2024-03-25T17:28:35.642 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:35.643 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T17:28:35.643 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T17:28:35.680 INFO:tasks.ceph.mon.b:Started 2024-03-25T17:28:35.680 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T17:28:35.681 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T17:28:35.681 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T17:28:35.683 INFO:tasks.ceph.mgr.y:Started 2024-03-25T17:28:35.683 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T17:28:35.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T17:28:35.685 INFO:tasks.ceph.mgr.x:Started 2024-03-25T17:28:35.686 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T17:28:35.686 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.786+0000 7fe112f7c640 10 monclient: get_monmap_and_config 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.786+0000 7fe112f7c640 10 monclient: build_initial_monmap 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.786+0000 7fe112f7c640 10 monclient: monmap: 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 0 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:28:35.786 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:35.786612+0000 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:35.786612+0000 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:28:35.787 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 1 Processor -- start 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 1 -- start start 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient: init 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient: _reopen_session rank -1 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c0a9920 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06b100 con 0x7fe10c0a9920 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b2940 con 0x7fe10c0b45a0 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c13fec0 con 0x7fe10c146450 2024-03-25T17:28:35.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:35.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe112f7c640 10 monclient(hunting): authenticate will time out at 5274.068773s 2024-03-25T17:28:35.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe10c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:35.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe10c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:35.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.787+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe10c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:35.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.988+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe10c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:35.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.988+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe10c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:35.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:35.988+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe10c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:36.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:36.389+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe10c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:36.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:36.389+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe10c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:36.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:36.389+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe10c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:37.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:37.190+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe10c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:37.190+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe10c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:37.190+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe10c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe10c14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:38.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe10c145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe10c13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c0a9920 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06b100 con 0x7fe10c0a9920 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b2940 con 0x7fe10c0b45a0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c13fec0 con 0x7fe10c146450 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe10bfff640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.787+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:38.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.788+0000 7fe110d18640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:38.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.989+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:38.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.989+0000 7fe10bfff640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:38.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:38.989+0000 7fe110d18640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:39.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:39.390+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:39.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:39.390+0000 7fe10bfff640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:39.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:39.390+0000 7fe110d18640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:40.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:40.191+0000 7fe111519640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:40.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:40.191+0000 7fe10bfff640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:40.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:40.191+0000 7fe110d18640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c0a9920 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0010e20 con 0x7fe10c0a9920 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06b100 con 0x7fe10c0b45a0 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b2940 con 0x7fe10c146450 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:41.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe111519640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:41.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.788+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:41.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.989+0000 7fe111519640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:41.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.989+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:41.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:41.989+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:42.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:42.389+0000 7fe111519640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:42.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:42.389+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:42.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:42.389+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:42.649 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:06.864 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:42.802 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:00:07.028 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:42.804 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:00:07.031 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:42.956 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:07.171 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:43.122 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:00:07.349 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:43.125 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:00:07.351 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:28:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:43.190+0000 7fe111519640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6789/0 2024-03-25T17:28:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:43.190+0000 7fe110d18640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.96:6790/0 2024-03-25T17:28:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:43.190+0000 7fe10bfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T17:28:44.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:44.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:44.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:44.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c0a9920 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c13fec0 con 0x7fe10c0a9920 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0010e20 con 0x7fe10c0b45a0 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06b100 con 0x7fe10c146450 2024-03-25T17:28:44.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.787+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:44.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.834+0000 7fe111519640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 0x7fe0f00056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:34850/0 (socket says 172.21.15.96:34850) 2024-03-25T17:28:44.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:44.834+0000 7fe111519640 1 -- 172.21.15.96:0/147877705 learned_addr learned my addr 172.21.15.96:0/147877705 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:28:47.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe10c0a9920 legacy=0x7fe0f0008e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f0005f60 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b2940 con 0x7fe0f0005f60 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0006cb0 con 0x7fe10c0b45a0 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0006e90 con 0x7fe10c146450 2024-03-25T17:28:47.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:47.787+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.787+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f0005f60 legacy=0x7fe0f00063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f0009610 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c146450 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00080d0 con 0x7fe0f0009610 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0009350 con 0x7fe10c0b45a0 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f000a2c0 con 0x7fe10c146450 2024-03-25T17:28:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:50.788+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:53.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:53.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe10c146450 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe10c0b45a0 legacy=0x7fe0f00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f0009610 legacy=0x7fe0f00099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f00064f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f00089b0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f0009610 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0006340 con 0x7fe0f0009610 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0008fc0 con 0x7fe0f00089b0 2024-03-25T17:28:53.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00099f0 con 0x7fe0f00064f0 2024-03-25T17:28:53.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:53.788+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:56.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:56.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:56.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:56.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f00064f0 legacy=0x7fe0f00082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe0f00089b0 legacy=0x7fe0f00160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f0009610 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f0009d10 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f000ce80 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f0009610 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0009b80 con 0x7fe0f0009610 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00082a0 con 0x7fe0f0009d10 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00084d0 con 0x7fe0f000ce80 2024-03-25T17:28:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:56.789+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 legacy=0x7fe0f000d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe0f0009d10 legacy=0x7fe0f0005650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f0009610 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:28:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f000dab0 addr v1:172.21.15.96:6789/0 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f00105b0 addr v1:172.21.15.203:6789/0 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f000ce80 addr v1:172.21.15.96:6790/0 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0010b80 con 0x7fe0f000dab0 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f0008660 con 0x7fe0f00105b0 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00087f0 con 0x7fe0f000ce80 2024-03-25T17:28:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.789+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:59.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.792+0000 7fe111519640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T17:28:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.792+0000 7fe111519640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:28:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.792+0000 7fe111519640 1 --1- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 0x7fe0f000c5f0 :-1 s=OPENED pgs=27 cs=1 l=1).handle_message read tag failed 2024-03-25T17:28:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.792+0000 7fe111519640 1 --1- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 0x7fe0f000c5f0 :-1 s=OPENED pgs=27 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:28:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:28:59.792+0000 7fe10b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.96:6790/0 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): tick 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe0f00105b0 legacy=0x7fe0f00170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f000dab0 legacy=0x7fe0f000fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:02.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f0005840 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f000ce80 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f0011090 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00056b0 con 0x7fe0f0005840 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f000de90 con 0x7fe0f0011090 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f000f080 con 0x7fe0f000ce80 2024-03-25T17:29:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:02.789+0000 7fe10a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898268730 0 0) 0x7fe0f00056b0 con 0x7fe0f0005840 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T17:29:03.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.661+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc003520 con 0x7fe0f0005840 2024-03-25T17:29:03.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529412360 0 0) 0x7fe0f000de90 con 0x7fe0f0011090 2024-03-25T17:29:03.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:03.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:03.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T17:29:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.693+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f00056b0 con 0x7fe0f0011090 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 536274659 0 0) 0x7fe0fc003520 con 0x7fe0f0005840 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6790/0 conn(0x7fe0f000ce80 legacy=0x7fe0f000c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.203:6789/0 conn(0x7fe0f0011090 legacy=0x7fe0f0013480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 monclient: found mon.noname-a 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:03.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00102c0 con 0x7fe0f0005840 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe100003f20 con 0x7fe0f0005840 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe10b7fe640 10 monclient: dump: 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe112f7c640 5 monclient: authenticate success, global_id 4101 2024-03-25T17:29:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.724+0000 7fe112f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe100004080 con 0x7fe0f0005840 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 1 -- 172.21.15.96:0/147877705 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe1000044d0 con 0x7fe0f0005840 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe10b7fe640 10 monclient: dump: 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.765 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe112f7c640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:03.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.764+0000 7fe112f7c640 4 set_mon_vals no callback set 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 10 monclient: shutdown 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 1 -- 172.21.15.96:0/147877705 >> v1:172.21.15.96:6789/0 conn(0x7fe0f0005840 legacy=0x7fe0f000d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 1 -- 172.21.15.96:0/147877705 shutdown_connections 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 1 -- 172.21.15.96:0/147877705 >> 172.21.15.96:0/147877705 conn(0x7fe10c0b7f40 msgr2=0x7fe10c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 1 -- 172.21.15.96:0/147877705 shutdown_connections 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 1 -- 172.21.15.96:0/147877705 wait complete. 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 10 monclient: build_initial_monmap 2024-03-25T17:29:03.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.765+0000 7fe112f7c640 10 monclient: monmap: 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 0 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:created 0.000000 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 librados: starting msgr at 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 librados: starting objecter 2024-03-25T17:29:03.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 Processor -- start 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 -- start start 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 librados: setting wanted keys 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 1 librados: calling monclient init 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.766+0000 7fe112f7c640 10 monclient: init 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f0005840 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f000ce80 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f0011090 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c143fe0 con 0x7fe0f000ce80 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c144470 con 0x7fe0f0011090 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c145310 con 0x7fe0f0005840 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.767+0000 7fe112f7c640 10 monclient(hunting): authenticate will time out at 5302.049358s 2024-03-25T17:29:03.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.768+0000 7fe110d18640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fe0f0005840 0x7fe10c143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:46440/0 (socket says 172.21.15.96:46440) 2024-03-25T17:29:03.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.768+0000 7fe110d18640 1 -- 172.21.15.96:0/4099461245 learned_addr learned my addr 172.21.15.96:0/4099461245 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:03.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.769+0000 7fe1097fa640 10 client.?.objecter ms_handle_connect 0x7fe0f0011090 2024-03-25T17:29:03.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.769+0000 7fe1097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:03.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.769+0000 7fe1097fa640 10 client.?.objecter ms_handle_connect 0x7fe0f0005840 2024-03-25T17:29:03.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.769+0000 7fe1097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:03.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.812+0000 7fe1097fa640 10 client.?.objecter ms_handle_connect 0x7fe0f000ce80 2024-03-25T17:29:03.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.812+0000 7fe1097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:03.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537440897 0 0) 0x7fe0f000f080 con 0x7fe0f000ce80 2024-03-25T17:29:03.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:03.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:03.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:03.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 10 monclient(hunting): my global_id is 4110 2024-03-25T17:29:03.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.817+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0005cc0 con 0x7fe0f000ce80 2024-03-25T17:29:03.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219961284 0 0) 0x7fe0f00056b0 con 0x7fe0f0011090 2024-03-25T17:29:03.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:03.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:03.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:03.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 10 monclient(hunting): my global_id is 4111 2024-03-25T17:29:03.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.824+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f000f080 con 0x7fe0f0011090 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142235183 0 0) 0x7fe0f0005cc0 con 0x7fe0f000ce80 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f00056b0 con 0x7fe0f000ce80 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe0f80041b0 con 0x7fe0f000ce80 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 10 monclient(hunting): dump: 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.834+0000 7fe1097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:03.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738890777 0 0) 0x7fe0f00102c0 con 0x7fe0f0005840 2024-03-25T17:29:03.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:03.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:03.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:03.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 10 monclient(hunting): my global_id is 4112 2024-03-25T17:29:03.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.835+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0005cc0 con 0x7fe0f0005840 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 466806947 0 0) 0x7fe0f00056b0 con 0x7fe0f000ce80 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 >> v1:172.21.15.96:6790/0 conn(0x7fe0f0005840 legacy=0x7fe10c143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 >> v1:172.21.15.203:6789/0 conn(0x7fe0f0011090 legacy=0x7fe10c1e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 10 monclient: _finish_hunting 0 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 1 monclient: found mon.a 2024-03-25T17:29:03.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c1455f0 con 0x7fe0f000ce80 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 10 monclient: _finish_auth 0 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 10 monclient: _check_auth_tickets 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe1097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 5 monclient: authenticate success, global_id 4110 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 10 monclient: _renew_subs 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c143fe0 con 0x7fe0f000ce80 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 10 monclient: _renew_subs 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:03.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c143fe0 con 0x7fe0f000ce80 2024-03-25T17:29:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.857+0000 7fe112f7c640 1 librados: init done 2024-03-25T17:29:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.858+0000 7fe112f7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.858+0000 7fe112f7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.858+0000 7fe112f7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:03.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:03.858+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c143fe0 con 0x7fe0f000ce80 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0f80045c0 con 0x7fe0f000ce80 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe0f80049d0 con 0x7fe0f000ce80 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 10 monclient: dump: 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.085 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe0f8004f80 con 0x7fe0f000ce80 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe10affd640 4 set_mon_vals no callback set 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 441186621 0 0) 0x7fe0f8006090 con 0x7fe0f000ce80 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 10 client.4110.objecter ms_dispatch 0x7fe10c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.083+0000 7fe1097fa640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T17:29:04.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.084+0000 7fe1097fa640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T17:29:04.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.084+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe0f8006820 con 0x7fe0f000ce80 2024-03-25T17:29:04.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.084+0000 7fe1097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:04.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.084+0000 7fe1097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.151+0000 7fe112f7c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.151+0000 7fe112f7c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:04.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.151+0000 7fe112f7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.151+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe10c143fe0 con 0x7fe0f000ce80 2024-03-25T17:29:04.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.330+0000 7fe1097fa640 1 -- 172.21.15.96:0/4099461245 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe0f8005540 con 0x7fe0f000ce80 2024-03-25T17:29:04.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.330+0000 7fe1097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:04.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.330+0000 7fe1097fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T17:29:04.331 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 10 librados: watch_flush enter 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 10 librados: watch_flush exit 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 10 client.4110.objecter successfully canceled tick 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 10 monclient: shutdown 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 >> v1:172.21.15.96:6789/0 conn(0x7fe0f000ce80 legacy=0x7fe10c1dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 shutdown_connections 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.331+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 >> 172.21.15.96:0/4099461245 conn(0x7fe10c0b7f40 msgr2=0x7fe10c0b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.332+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 shutdown_connections 2024-03-25T17:29:04.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.332+0000 7fe112f7c640 1 -- 172.21.15.96:0/4099461245 wait complete. 2024-03-25T17:29:04.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:04.332+0000 7fe112f7c640 1 librados: shutdown 2024-03-25T17:29:04.343 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T17:29:04.343 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T17:29:04.343 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:29:04.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T17:29:04.371 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:29:04.372 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T17:29:04.408 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:04.408+0000 7f804442b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:29:04.440 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:29:04.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T17:29:04.461 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:04.461+0000 7f804442b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:29:04.508 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-25T17:29:04.508 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T17:29:04.513 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:04.512+0000 7fcbb22c3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:29:04.539 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:29:04.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T17:29:04.566 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:29:04.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T17:29:04.572 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:04.572+0000 7fcbb22c3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:29:04.633 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:29:04.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T17:29:04.702 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:29:04.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T17:29:04.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new c67c12af-45d9-4991-a443-4e6b4ce66e07 0 2024-03-25T17:29:04.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 10 monclient: get_monmap_and_config 2024-03-25T17:29:04.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 10 monclient: build_initial_monmap 2024-03-25T17:29:04.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 10 monclient: monmap: 2024-03-25T17:29:04.913 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:04.913620+0000 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:04.913620+0000 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:04.914 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 1 Processor -- start 2024-03-25T17:29:04.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 1 -- start start 2024-03-25T17:29:04.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.913+0000 7f4179d6b640 10 monclient: init 2024-03-25T17:29:04.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:04.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f41740b4700 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f4174142a20 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f41740a96b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41740a9d60 con 0x7f4174142a20 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417413ff10 con 0x7f41740b4700 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41740aa1f0 con 0x7f41740a96b0 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.914+0000 7f4179d6b640 10 monclient(hunting): authenticate will time out at 5295.222175s 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.915+0000 7f4172ffd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4174142a20 0x7f417413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58322/0 (socket says 172.21.15.203:58322) 2024-03-25T17:29:04.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.915+0000 7f4172ffd640 1 -- 172.21.15.203:0/978569150 learned_addr learned my addr 172.21.15.203:0/978569150 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031371223 0 0) 0x7f41740a9d60 con 0x7f4174142a20 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T17:29:04.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.917+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c003520 con 0x7f4174142a20 2024-03-25T17:29:04.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339959454 0 0) 0x7f415c003520 con 0x7f4174142a20 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 >> v1:172.21.15.96:6790/0 conn(0x7f41740a96b0 legacy=0x7f417413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 >> v1:172.21.15.203:6789/0 conn(0x7f41740b4700 legacy=0x7f4174142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 monclient: found mon.noname-a 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740b4ae0 con 0x7f4174142a20 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4168001690 con 0x7f4174142a20 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f41727fc640 10 monclient: dump: 2024-03-25T17:29:04.921 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f4179d6b640 5 monclient: authenticate success, global_id 4116 2024-03-25T17:29:04.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.920+0000 7f4179d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:04.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4168001860 con 0x7f4174142a20 2024-03-25T17:29:04.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 1 -- 172.21.15.203:0/978569150 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4168001cb0 con 0x7f4174142a20 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f4179d6b640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f4179d6b640 4 set_mon_vals no callback set 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f41727fc640 10 monclient: dump: 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.924 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.922+0000 7f4179d6b640 10 monclient: shutdown 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 -- 172.21.15.203:0/978569150 >> v1:172.21.15.96:6789/0 conn(0x7f4174142a20 legacy=0x7f417413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 -- 172.21.15.203:0/978569150 shutdown_connections 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 -- 172.21.15.203:0/978569150 >> 172.21.15.203:0/978569150 conn(0x7f41740668e0 msgr2=0x7f4174066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 -- 172.21.15.203:0/978569150 shutdown_connections 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 -- 172.21.15.203:0/978569150 wait complete. 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 10 monclient: build_initial_monmap 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 10 monclient: monmap: 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:04.925 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 librados: starting msgr at 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.923+0000 7f4179d6b640 1 librados: starting objecter 2024-03-25T17:29:04.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 Processor -- start 2024-03-25T17:29:04.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 -- start start 2024-03-25T17:29:04.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 librados: setting wanted keys 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 librados: calling monclient init 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient: init 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f41740a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f41740b4700 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f4174142a20 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41741db070 con 0x7f4174142a20 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41741db200 con 0x7f41740a96b0 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4174135630 con 0x7f41740b4700 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.924+0000 7f4179d6b640 10 monclient(hunting): authenticate will time out at 5295.232497s 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.925+0000 7f4173fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4174142a20 0x7f41741d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58328/0 (socket says 172.21.15.203:58328) 2024-03-25T17:29:04.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.925+0000 7f4173fff640 1 -- 172.21.15.203:0/137380805 learned_addr learned my addr 172.21.15.203:0/137380805 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.925+0000 7f415bfff640 10 client.?.objecter ms_handle_connect 0x7f4174142a20 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.925+0000 7f415bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679667937 0 0) 0x7f41741db070 con 0x7f4174142a20 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 10 monclient(hunting): my global_id is 4119 2024-03-25T17:29:04.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.927+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4160004eb0 con 0x7f4174142a20 2024-03-25T17:29:04.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.932+0000 7f415bfff640 10 client.?.objecter ms_handle_connect 0x7f41740b4700 2024-03-25T17:29:04.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.932+0000 7f415bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362135439 0 0) 0x7f4174135630 con 0x7f41740b4700 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 10 monclient(hunting): my global_id is 4118 2024-03-25T17:29:04.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.934+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41741db070 con 0x7f41740b4700 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885799379 0 0) 0x7f4160004eb0 con 0x7f4174142a20 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4174135630 con 0x7f4174142a20 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f416c001c60 con 0x7f4174142a20 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 10 monclient(hunting): dump: 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.938 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.938+0000 7f415bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:04.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553257781 0 0) 0x7f41741db070 con 0x7f41740b4700 2024-03-25T17:29:04.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4160004eb0 con 0x7f41740b4700 2024-03-25T17:29:04.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4168001b70 con 0x7f41740b4700 2024-03-25T17:29:04.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.940+0000 7f415bfff640 10 monclient(hunting): dump: 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.941 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4191841953 0 0) 0x7f4174135630 con 0x7f4174142a20 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 >> v1:172.21.15.96:6790/0 conn(0x7f41740b4700 legacy=0x7f41741d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 >> v1:172.21.15.203:6789/0 conn(0x7f41740a96b0 legacy=0x7f41741da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 1 monclient: found mon.a 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41741357c0 con 0x7f4174142a20 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 10 monclient: _finish_auth 0 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f415bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 5 monclient: authenticate success, global_id 4119 2024-03-25T17:29:04.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 10 monclient: _renew_subs 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41741e2eb0 con 0x7f4174142a20 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 10 monclient: _renew_subs 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41741e3220 con 0x7f4174142a20 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.941+0000 7f4179d6b640 1 librados: init done 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.942+0000 7f4179d6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.942+0000 7f4179d6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.942+0000 7f4179d6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:04.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.942+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417404f730 con 0x7f4174142a20 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f416c003970 con 0x7f4174142a20 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f416c003cc0 con 0x7f4174142a20 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 10 monclient: dump: 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:04.948 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f4171ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.948+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f416c004270 con 0x7f4174142a20 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.949+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2079309483 0 0) 0x7f416c006830 con 0x7f4174142a20 2024-03-25T17:29:04.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.949+0000 7f415bfff640 10 client.4119.objecter ms_dispatch 0x7f4174000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T17:29:04.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.949+0000 7f415bfff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T17:29:04.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.949+0000 7f415bfff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T17:29:04.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:04.949+0000 7f415bfff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T17:29:05.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.040+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f416c004520 con 0x7f4174142a20 2024-03-25T17:29:05.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.041+0000 7f415bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.041+0000 7f415bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:05.053 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.053+0000 7f804442b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:29:05.110 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.110+0000 7f804442b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:29:05.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.114+0000 7f4179d6b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c67c12af-45d9-4991-a443-4e6b4ce66e07", "id": 0}] 2024-03-25T17:29:05.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.114+0000 7f4179d6b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c67c12af-45d9-4991-a443-4e6b4ce66e07", "id": 0}] 2024-03-25T17:29:05.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.114+0000 7f4179d6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.114+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c67c12af-45d9-4991-a443-4e6b4ce66e07", "id": 0} v 0) v1 -- 0x7f41741db390 con 0x7f4174142a20 2024-03-25T17:29:05.137 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.137+0000 7fcbb22c3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:29:05.155 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.156+0000 7f804442b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:29:05.194 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.194+0000 7fcbb22c3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:29:05.219 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.219+0000 7f415bfff640 1 -- 172.21.15.203:0/137380805 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c67c12af-45d9-4991-a443-4e6b4ce66e07", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1348727481 0 732707515) 0x7f416c004760 con 0x7f4174142a20 2024-03-25T17:29:05.219 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.219+0000 7f415bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c67c12af-45d9-4991-a443-4e6b4ce66e07", "id": 0}] 2024-03-25T17:29:05.219 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.219+0000 7f415bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:05.219 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 10 librados: watch_flush enter 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 10 librados: watch_flush exit 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 10 client.4119.objecter successfully canceled tick 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 10 monclient: shutdown 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 >> v1:172.21.15.96:6789/0 conn(0x7f4174142a20 legacy=0x7f41741d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 shutdown_connections 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 >> 172.21.15.203:0/137380805 conn(0x7f41740668e0 msgr2=0x7f417413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 shutdown_connections 2024-03-25T17:29:05.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 -- 172.21.15.203:0/137380805 wait complete. 2024-03-25T17:29:05.222 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.220+0000 7f4179d6b640 1 librados: shutdown 2024-03-25T17:29:05.224 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.224+0000 7f804442b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:29:05.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new ad03c452-1e7a-4d1f-bbd6-b3c737a5e354 1 2024-03-25T17:29:05.255 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.255+0000 7fcbb22c3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:29:05.287 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.287+0000 7f804442b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:29:05.313 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.312+0000 7fcbb22c3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 10 monclient: get_monmap_and_config 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 10 monclient: build_initial_monmap 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 10 monclient: monmap: 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:05.336528+0000 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:05.336528+0000 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:05.337 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 1 Processor -- start 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 1 -- start start 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.336+0000 7f885db96640 10 monclient: init 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient(hunting): picked mon.noname-b con 0x7f88580a9710 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient(hunting): picked mon.noname-c con 0x7f88580aafa0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient(hunting): picked mon.noname-a con 0x7f885813c570 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580ab420 con 0x7f885813c570 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b0190 con 0x7f88580aafa0 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580abdb0 con 0x7f88580a9710 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.337+0000 7f885db96640 10 monclient(hunting): authenticate will time out at 5295.645118s 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.338+0000 7f8856ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f88580aafa0 0x7f88580724a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47082/0 (socket says 172.21.15.203:47082) 2024-03-25T17:29:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.338+0000 7f8856ffd640 1 -- 172.21.15.203:0/41304177 learned_addr learned my addr 172.21.15.203:0/41304177 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:05.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501792748 0 0) 0x7f88580abdb0 con 0x7f88580a9710 2024-03-25T17:29:05.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8844003520 con 0x7f88580a9710 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767810848 0 0) 0x7f88580ab420 con 0x7f885813c570 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580abdb0 con 0x7f885813c570 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963241661 0 0) 0x7f88580b0190 con 0x7f88580aafa0 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T17:29:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.341+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580ab420 con 0x7f88580aafa0 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3471163626 0 0) 0x7f88580abdb0 con 0x7f885813c570 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 >> v1:172.21.15.96:6790/0 conn(0x7f88580a9710 legacy=0x7f88580ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 >> v1:172.21.15.203:6789/0 conn(0x7f88580aafa0 legacy=0x7f88580724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 monclient: found mon.noname-a 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580abf40 con 0x7f885813c570 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8850001670 con 0x7f885813c570 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f88567fc640 10 monclient: dump: 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f885db96640 5 monclient: authenticate success, global_id 4122 2024-03-25T17:29:05.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.344+0000 7f885db96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88500017d0 con 0x7f885813c570 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 1 -- 172.21.15.203:0/41304177 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8850001be0 con 0x7f885813c570 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f885db96640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f885db96640 4 set_mon_vals no callback set 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f88567fc640 10 monclient: dump: 2024-03-25T17:29:05.348 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f885db96640 10 monclient: shutdown 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f885db96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.346+0000 7f885db96640 1 -- 172.21.15.203:0/41304177 >> v1:172.21.15.96:6789/0 conn(0x7f885813c570 legacy=0x7f885814a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 1 -- 172.21.15.203:0/41304177 shutdown_connections 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 1 -- 172.21.15.203:0/41304177 >> 172.21.15.203:0/41304177 conn(0x7f885806b560 msgr2=0x7f88580b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:05.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 1 -- 172.21.15.203:0/41304177 shutdown_connections 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 1 -- 172.21.15.203:0/41304177 wait complete. 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 10 monclient: build_initial_monmap 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.347+0000 7f885db96640 10 monclient: monmap: 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 librados: starting msgr at 2024-03-25T17:29:05.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 librados: starting objecter 2024-03-25T17:29:05.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 Processor -- start 2024-03-25T17:29:05.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 -- start start 2024-03-25T17:29:05.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 librados: setting wanted keys 2024-03-25T17:29:05.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 librados: calling monclient init 2024-03-25T17:29:05.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient: init 2024-03-25T17:29:05.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient(hunting): picked mon.noname-b con 0x7f88580a9710 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient(hunting): picked mon.noname-a con 0x7f88580aafa0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient(hunting): picked mon.noname-c con 0x7f885813c570 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88581399c0 con 0x7f88580aafa0 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8858138e70 con 0x7f88580a9710 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885813cf00 con 0x7f885813c570 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.348+0000 7f885db96640 10 monclient(hunting): authenticate will time out at 5295.656562s 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.349+0000 7f88577fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f88580a9710 0x7f8858138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47096/0 (socket says 172.21.15.203:47096) 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.349+0000 7f88577fe640 1 -- 172.21.15.203:0/2511778015 learned_addr learned my addr 172.21.15.203:0/2511778015 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.349+0000 7f884ffff640 10 client.?.objecter ms_handle_connect 0x7f88580a9710 2024-03-25T17:29:05.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.349+0000 7f884ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.350+0000 7f884ffff640 10 client.?.objecter ms_handle_connect 0x7f885813c570 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.350+0000 7f884ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.350+0000 7f884ffff640 10 client.?.objecter ms_handle_connect 0x7f88580aafa0 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.350+0000 7f884ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451506594 0 0) 0x7f88581399c0 con 0x7f88580aafa0 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): my global_id is 4125 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8838004eb0 con 0x7f88580aafa0 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798067607 0 0) 0x7f885813cf00 con 0x7f885813c570 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T17:29:05.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88581399c0 con 0x7f885813c570 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264670614 0 0) 0x7f8858138e70 con 0x7f88580a9710 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 10 monclient(hunting): my global_id is 4120 2024-03-25T17:29:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.352+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f885813cf00 con 0x7f88580a9710 2024-03-25T17:29:05.356 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.356+0000 7fcbb22c3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:29:05.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689644797 0 0) 0x7f8838004eb0 con 0x7f88580aafa0 2024-03-25T17:29:05.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8858138e70 con 0x7f88580aafa0 2024-03-25T17:29:05.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f88400018f0 con 0x7f88580aafa0 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 10 monclient(hunting): dump: 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:05.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406992099 0 0) 0x7f88581399c0 con 0x7f885813c570 2024-03-25T17:29:05.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.357+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8838004eb0 con 0x7f885813c570 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774112583 0 0) 0x7f885813cf00 con 0x7f88580a9710 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88581399c0 con 0x7f88580a9710 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8848001c90 con 0x7f88580a9710 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 10 monclient(hunting): dump: 2024-03-25T17:29:05.360 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.359+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1706339631 0 0) 0x7f8858138e70 con 0x7f88580aafa0 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 >> v1:172.21.15.96:6790/0 conn(0x7f885813c570 legacy=0x7f88580b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 >> v1:172.21.15.203:6789/0 conn(0x7f88580a9710 legacy=0x7f8858138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 1 monclient: found mon.a 2024-03-25T17:29:05.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88581378b0 con 0x7f88580aafa0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 10 monclient: _finish_auth 0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f884ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 5 monclient: authenticate success, global_id 4125 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _renew_subs 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8858137b60 con 0x7f88580aafa0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _renew_subs 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88581dee40 con 0x7f88580aafa0 2024-03-25T17:29:05.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 1 librados: init done 2024-03-25T17:29:05.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.360+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88580ad630 con 0x7f88580aafa0 2024-03-25T17:29:05.417 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.417+0000 7f804442b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8840001d00 con 0x7f88580aafa0 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8840004980 con 0x7f88580aafa0 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 10 monclient: dump: 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.461 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8840004f30 con 0x7f88580aafa0 2024-03-25T17:29:05.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.461+0000 7f8855ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1709318595 0 0) 0x7f88400064f0 con 0x7f88580aafa0 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 10 client.4125.objecter ms_dispatch 0x7f8858000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8840005410 con 0x7f88580aafa0 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.462+0000 7f884ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:05.477 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.477+0000 7fcbb22c3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:29:05.490 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.490+0000 7f804442b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:05.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.536+0000 7f885db96640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ad03c452-1e7a-4d1f-bbd6-b3c737a5e354", "id": 1}] 2024-03-25T17:29:05.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.536+0000 7f885db96640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ad03c452-1e7a-4d1f-bbd6-b3c737a5e354", "id": 1}] 2024-03-25T17:29:05.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.536+0000 7f885db96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.536+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ad03c452-1e7a-4d1f-bbd6-b3c737a5e354", "id": 1} v 0) v1 -- 0x7f885804f730 con 0x7f88580aafa0 2024-03-25T17:29:05.539 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.539+0000 7f804442b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:29:05.548 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.548+0000 7fcbb22c3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:05.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.589+0000 7f884ffff640 1 -- 172.21.15.203:0/2511778015 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad03c452-1e7a-4d1f-bbd6-b3c737a5e354", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3314548271 0 940483276) 0x7f8840005870 con 0x7f88580aafa0 2024-03-25T17:29:05.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.589+0000 7f884ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ad03c452-1e7a-4d1f-bbd6-b3c737a5e354", "id": 1}] 2024-03-25T17:29:05.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.589+0000 7f884ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:05.589 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 10 librados: watch_flush enter 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 10 librados: watch_flush exit 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 10 client.4125.objecter successfully canceled tick 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 10 monclient: shutdown 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 >> v1:172.21.15.96:6789/0 conn(0x7f88580aafa0 legacy=0x7f88580b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 shutdown_connections 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 >> 172.21.15.203:0/2511778015 conn(0x7f885806b560 msgr2=0x7f8858074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:05.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 shutdown_connections 2024-03-25T17:29:05.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 -- 172.21.15.203:0/2511778015 wait complete. 2024-03-25T17:29:05.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.590+0000 7f885db96640 1 librados: shutdown 2024-03-25T17:29:05.599 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.598+0000 7fcbb22c3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:29:05.600 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.601+0000 7f804442b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:29:05.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new e5d86866-0c9f-4715-90ed-4761036cb0b0 2 2024-03-25T17:29:05.651 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.651+0000 7fcbb22c3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:29:05.661 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:05.661+0000 7f804442b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:29:05.698 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:05.697+0000 7fcbb22c3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.703+0000 7f59a3baf640 10 monclient: get_monmap_and_config 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.703+0000 7f59a3baf640 10 monclient: build_initial_monmap 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.703+0000 7f59a3baf640 10 monclient: monmap: 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:05.703697+0000 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:05.703697+0000 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:05.703 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:05.704 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:05.704 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.703+0000 7f59a3baf640 1 Processor -- start 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 1 -- start start 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient: init 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:05.705 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-a con 0x7f599c0b4700 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-c con 0x7f599c142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-b con 0x7f599c0a96b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0a9d60 con 0x7f599c0b4700 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c13ff10 con 0x7f599c142a20 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0aa1f0 con 0x7f599c0a96b0 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.704+0000 7f59a3baf640 10 monclient(hunting): authenticate will time out at 5296.012252s 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.705+0000 7f59a194b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f599c0b4700 0x7f599c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58364/0 (socket says 172.21.15.203:58364) 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.705+0000 7f59a194b640 1 -- 172.21.15.203:0/1395375029 learned_addr learned my addr 172.21.15.203:0/1395375029 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:05.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.705+0000 7f59a114a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f599c142a20 0x7f599c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47100/0 (socket says 172.21.15.203:47100) 2024-03-25T17:29:05.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287699434 0 0) 0x7f599c0a9d60 con 0x7f599c0b4700 2024-03-25T17:29:05.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): my global_id is 4128 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003520 con 0x7f599c0b4700 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736362045 0 0) 0x7f599c13ff10 con 0x7f599c142a20 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 10 monclient(hunting): my global_id is 4123 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.707+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c0a9d60 con 0x7f599c142a20 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196340339 0 0) 0x7f599c0aa1f0 con 0x7f599c0a96b0 2024-03-25T17:29:05.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 10 monclient(hunting): my global_id is 4127 2024-03-25T17:29:05.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.708+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c13ff10 con 0x7f599c0a96b0 2024-03-25T17:29:05.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755290537 0 0) 0x7f599c0a9d60 con 0x7f599c142a20 2024-03-25T17:29:05.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 >> v1:172.21.15.96:6790/0 conn(0x7f599c0a96b0 legacy=0x7f599c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 >> v1:172.21.15.96:6789/0 conn(0x7f599c0b4700 legacy=0x7f599c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: _finish_hunting 0 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 monclient: found mon.noname-c 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0b4ae0 con 0x7f599c142a20 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: _finish_auth 0 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: _check_auth_tickets 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f598c001670 con 0x7f599c142a20 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a3baf640 5 monclient: authenticate success, global_id 4123 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a0949640 10 monclient: dump: 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.710+0000 7f59a3baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:05.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f598c0017d0 con 0x7f599c142a20 2024-03-25T17:29:05.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:05.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a3baf640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:05.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a3baf640 4 set_mon_vals no callback set 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 1 -- 172.21.15.203:0/1395375029 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f598c001be0 con 0x7f599c142a20 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.712+0000 7f59a0949640 10 monclient: dump: 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 10 monclient: shutdown 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 1 -- 172.21.15.203:0/1395375029 >> v1:172.21.15.203:6789/0 conn(0x7f599c142a20 legacy=0x7f599c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 1 -- 172.21.15.203:0/1395375029 shutdown_connections 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 1 -- 172.21.15.203:0/1395375029 >> 172.21.15.203:0/1395375029 conn(0x7f599c0668e0 msgr2=0x7f599c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 1 -- 172.21.15.203:0/1395375029 shutdown_connections 2024-03-25T17:29:05.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.713+0000 7f59a3baf640 1 -- 172.21.15.203:0/1395375029 wait complete. 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 10 monclient: build_initial_monmap 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 10 monclient: monmap: 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:05.717 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 librados: starting msgr at 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 librados: starting objecter 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 Processor -- start 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 -- start start 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 librados: setting wanted keys 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 1 librados: calling monclient init 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.714+0000 7f59a3baf640 10 monclient: init 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-a con 0x7f599c0a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-c con 0x7f599c0b4700 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient(hunting): picked mon.noname-b con 0x7f599c142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c13c880 con 0x7f599c0a96b0 2024-03-25T17:29:05.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0ac300 con 0x7f599c142a20 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c13c510 con 0x7f599c0b4700 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.715+0000 7f59a3baf640 10 monclient(hunting): authenticate will time out at 5296.023291s 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59a194b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f599c0a96b0 0x7f599c13be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58374/0 (socket says 172.21.15.203:58374) 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59a194b640 1 -- 172.21.15.203:0/78101847 learned_addr learned my addr 172.21.15.203:0/78101847 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter ms_handle_connect 0x7f599c0b4700 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter ms_handle_connect 0x7f599c0a96b0 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter ms_handle_connect 0x7f599c142a20 2024-03-25T17:29:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.716+0000 7f59927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241806894 0 0) 0x7f599c13c880 con 0x7f599c0a96b0 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5980004eb0 con 0x7f599c0a96b0 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727409041 0 0) 0x7f599c13c510 con 0x7f599c0b4700 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): my global_id is 4130 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c13c880 con 0x7f599c0b4700 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673910132 0 0) 0x7f599c0ac300 con 0x7f599c142a20 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T17:29:05.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.718+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c13c510 con 0x7f599c142a20 2024-03-25T17:29:05.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273693662 0 0) 0x7f5980004eb0 con 0x7f599c0a96b0 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f599c0ac300 con 0x7f599c0a96b0 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5998001950 con 0x7f599c0a96b0 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): dump: 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.724 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644764860 0 0) 0x7f599c13c880 con 0x7f599c0b4700 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5980004eb0 con 0x7f599c0b4700 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f598c001ae0 con 0x7f599c0b4700 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): dump: 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.725 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134135636 0 0) 0x7f599c13c510 con 0x7f599c142a20 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f599c13c880 con 0x7f599c142a20 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5994001c70 con 0x7f599c142a20 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.723+0000 7f59927fc640 10 monclient(hunting): dump: 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.726 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.727 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.727 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.727 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.727 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 870965415 0 0) 0x7f599c0ac300 con 0x7f599c0a96b0 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b4700 legacy=0x7f599c0ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 >> v1:172.21.15.203:6789/0 conn(0x7f599c142a20 legacy=0x7f599c1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 1 monclient: found mon.a 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0acea0 con 0x7f599c0a96b0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 5 monclient: authenticate success, global_id 4131 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 10 monclient: _renew_subs 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c13b260 con 0x7f599c0a96b0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 10 monclient: _renew_subs 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c139d70 con 0x7f599c0a96b0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.726+0000 7f59a3baf640 1 librados: init done 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.727+0000 7f59a3baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.727+0000 7f59a3baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.727+0000 7f59a3baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.727+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c04f730 con 0x7f599c0a96b0 2024-03-25T17:29:05.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59980047e0 con 0x7f599c0a96b0 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5998004af0 con 0x7f599c0a96b0 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 10 monclient: dump: 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f5993fff640 4 set_mon_vals no callback set 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5998004fe0 con 0x7f599c0a96b0 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1694216044 0 0) 0x7f5998006650 con 0x7f599c0a96b0 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 10 client.4131.objecter ms_dispatch 0x7f599c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T17:29:05.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.730+0000 7f59927fc640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T17:29:05.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.824+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5998006320 con 0x7f599c0a96b0 2024-03-25T17:29:05.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.824+0000 7f59927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:05.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.824+0000 7f59927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.938+0000 7f59a3baf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e5d86866-0c9f-4715-90ed-4761036cb0b0", "id": 2}] 2024-03-25T17:29:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.938+0000 7f59a3baf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e5d86866-0c9f-4715-90ed-4761036cb0b0", "id": 2}] 2024-03-25T17:29:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.938+0000 7f59a3baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.938+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e5d86866-0c9f-4715-90ed-4761036cb0b0", "id": 2} v 0) v1 -- 0x7f599c13fc90 con 0x7f599c0a96b0 2024-03-25T17:29:05.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.998+0000 7f59927fc640 1 -- 172.21.15.203:0/78101847 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e5d86866-0c9f-4715-90ed-4761036cb0b0", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 57808706 0 216599125) 0x7f5998006880 con 0x7f599c0a96b0 2024-03-25T17:29:05.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.998+0000 7f59927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e5d86866-0c9f-4715-90ed-4761036cb0b0", "id": 2}] 2024-03-25T17:29:05.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.998+0000 7f59927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:05.998 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 10 librados: watch_flush enter 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 10 librados: watch_flush exit 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 10 client.4131.objecter successfully canceled tick 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 10 monclient: shutdown 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 >> v1:172.21.15.96:6789/0 conn(0x7f599c0a96b0 legacy=0x7f599c13be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 shutdown_connections 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 >> 172.21.15.203:0/78101847 conn(0x7f599c0668e0 msgr2=0x7f599c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 shutdown_connections 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 -- 172.21.15.203:0/78101847 wait complete. 2024-03-25T17:29:06.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:05.999+0000 7f59a3baf640 1 librados: shutdown 2024-03-25T17:29:06.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 318e04fa-02d7-4358-879a-2613abedef73 3 2024-03-25T17:29:06.093 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.093+0000 7f804442b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:29:06.098 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.098+0000 7fcbb22c3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 10 monclient: get_monmap_and_config 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 10 monclient: build_initial_monmap 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 10 monclient: monmap: 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:06.112575+0000 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:06.112575+0000 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:06.112 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 1 Processor -- start 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 1 -- start start 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.112+0000 7fcb78f9d640 10 monclient: init 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb740b4700 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb74142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb740a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740a9d60 con 0x7fcb740a96b0 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7413ff10 con 0x7fcb74142a20 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740aa1f0 con 0x7fcb740b4700 2024-03-25T17:29:06.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:06.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.113+0000 7fcb78f9d640 10 monclient(hunting): authenticate will time out at 5296.421179s 2024-03-25T17:29:06.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.114+0000 7fcb72d9d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcb740a96b0 0x7fcb7413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58380/0 (socket says 172.21.15.203:58380) 2024-03-25T17:29:06.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.114+0000 7fcb72d9d640 1 -- 172.21.15.203:0/1591881642 learned_addr learned my addr 172.21.15.203:0/1591881642 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:06.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543033529 0 0) 0x7fcb740a9d60 con 0x7fcb740a96b0 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): my global_id is 4134 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c003520 con 0x7fcb740a96b0 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588453776 0 0) 0x7fcb7413ff10 con 0x7fcb74142a20 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): my global_id is 4129 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb740a9d60 con 0x7fcb74142a20 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127352046 0 0) 0x7fcb740aa1f0 con 0x7fcb740b4700 2024-03-25T17:29:06.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 10 monclient(hunting): my global_id is 4133 2024-03-25T17:29:06.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.116+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7413ff10 con 0x7fcb740b4700 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377566938 0 0) 0x7fcb5c003520 con 0x7fcb740a96b0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 >> v1:172.21.15.96:6790/0 conn(0x7fcb740b4700 legacy=0x7fcb74142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 >> v1:172.21.15.203:6789/0 conn(0x7fcb74142a20 legacy=0x7fcb7413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: _finish_hunting 0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 monclient: found mon.noname-a 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740b4ae0 con 0x7fcb740a96b0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: _finish_auth 0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: _check_auth_tickets 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb6c001a10 con 0x7fcb740a96b0 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb78f9d640 5 monclient: authenticate success, global_id 4134 2024-03-25T17:29:06.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb78f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb7159a640 10 monclient: dump: 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.121 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.119+0000 7fcb78f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:06.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb6c001b70 con 0x7fcb740a96b0 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 1 -- 172.21.15.203:0/1591881642 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb6c001f80 con 0x7fcb740a96b0 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb7159a640 10 monclient: dump: 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.123 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb78f9d640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.121+0000 7fcb78f9d640 4 set_mon_vals no callback set 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 10 monclient: shutdown 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 1 -- 172.21.15.203:0/1591881642 >> v1:172.21.15.96:6789/0 conn(0x7fcb740a96b0 legacy=0x7fcb7413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 1 -- 172.21.15.203:0/1591881642 shutdown_connections 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 1 -- 172.21.15.203:0/1591881642 >> 172.21.15.203:0/1591881642 conn(0x7fcb740668e0 msgr2=0x7fcb74066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 1 -- 172.21.15.203:0/1591881642 shutdown_connections 2024-03-25T17:29:06.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.122+0000 7fcb78f9d640 1 -- 172.21.15.203:0/1591881642 wait complete. 2024-03-25T17:29:06.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 10 monclient: build_initial_monmap 2024-03-25T17:29:06.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 10 monclient: monmap: 2024-03-25T17:29:06.126 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:06.126 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 librados: starting msgr at 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 librados: starting objecter 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 Processor -- start 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 -- start start 2024-03-25T17:29:06.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 librados: setting wanted keys 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 1 librados: calling monclient init 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.123+0000 7fcb78f9d640 10 monclient: init 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb740a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb740b4700 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb74142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740aba20 con 0x7fcb740a96b0 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740abd00 con 0x7fcb74142a20 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7413bd80 con 0x7fcb740b4700 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.124+0000 7fcb78f9d640 10 monclient(hunting): authenticate will time out at 5296.432123s 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb7259c640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcb740a96b0 0x7fcb740ac120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58382/0 (socket says 172.21.15.203:58382) 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb7259c640 1 -- 172.21.15.203:0/3196653749 learned_addr learned my addr 172.21.15.203:0/3196653749 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter ms_handle_connect 0x7fcb74142a20 2024-03-25T17:29:06.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter ms_handle_connect 0x7fcb740a96b0 2024-03-25T17:29:06.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter ms_handle_connect 0x7fcb740b4700 2024-03-25T17:29:06.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.125+0000 7fcb62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127270919 0 0) 0x7fcb740aba20 con 0x7fcb740a96b0 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb54004eb0 con 0x7fcb740a96b0 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578432483 0 0) 0x7fcb7413bd80 con 0x7fcb740b4700 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): my global_id is 4136 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb740aba20 con 0x7fcb740b4700 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024604558 0 0) 0x7fcb740abd00 con 0x7fcb74142a20 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T17:29:06.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.127+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7413bd80 con 0x7fcb74142a20 2024-03-25T17:29:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 293974357 0 0) 0x7fcb54004eb0 con 0x7fcb740a96b0 2024-03-25T17:29:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb740abd00 con 0x7fcb740a96b0 2024-03-25T17:29:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb64001a10 con 0x7fcb740a96b0 2024-03-25T17:29:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674450990 0 0) 0x7fcb740aba20 con 0x7fcb740b4700 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb54004eb0 con 0x7fcb740b4700 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb680018b0 con 0x7fcb740b4700 2024-03-25T17:29:06.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.132+0000 7fcb62ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.134 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963398116 0 0) 0x7fcb7413bd80 con 0x7fcb74142a20 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb740aba20 con 0x7fcb74142a20 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb6c001e80 con 0x7fcb74142a20 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.133+0000 7fcb62ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.136 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.135+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469916876 0 0) 0x7fcb54004eb0 con 0x7fcb740b4700 2024-03-25T17:29:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 >> v1:172.21.15.203:6789/0 conn(0x7fcb74142a20 legacy=0x7fcb741e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 >> v1:172.21.15.96:6789/0 conn(0x7fcb740a96b0 legacy=0x7fcb740ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 1 monclient: found mon.c 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7413ccb0 con 0x7fcb740b4700 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 5 monclient: authenticate success, global_id 4136 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 monclient: _renew_subs 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7413ac10 con 0x7fcb740b4700 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 monclient: _renew_subs 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7413b8d0 con 0x7fcb740b4700 2024-03-25T17:29:06.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 1 librados: init done 2024-03-25T17:29:06.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.136+0000 7fcb78f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.137+0000 7fcb78f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.137+0000 7fcb78f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:06.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.137+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7413bf10 con 0x7fcb740b4700 2024-03-25T17:29:06.233 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.233+0000 7f804442b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:29:06.238 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.237+0000 7fcbb22c3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:29:06.346 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.346+0000 7fcbb22c3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb680035f0 con 0x7fcb740b4700 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcb68001d80 con 0x7fcb740b4700 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 10 monclient: dump: 2024-03-25T17:29:06.348 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcb68003d10 con 0x7fcb740b4700 2024-03-25T17:29:06.349 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.348+0000 7fcb70d99640 4 set_mon_vals no callback set 2024-03-25T17:29:06.350 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.350+0000 7f804442b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.350+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3578664425 0 0) 0x7fcb680053e0 con 0x7fcb740b4700 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.350+0000 7fcb62ffd640 10 client.4136.objecter ms_dispatch 0x7fcb74000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.350+0000 7fcb62ffd640 3 client.4136.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.350+0000 7fcb62ffd640 3 client.4136.objecter handle_osd_map decoding full epoch 5 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.350+0000 7fcb62ffd640 20 client.4136.objecter dump_active .. 0 homeless 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.351+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcb68004f70 con 0x7fcb740b4700 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.351+0000 7fcb62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.351+0000 7fcb62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:06.392 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.391+0000 7fcbb22c3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:29:06.397 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.397+0000 7f804442b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:29:06.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.421+0000 7fcb78f9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "318e04fa-02d7-4358-879a-2613abedef73", "id": 3}] 2024-03-25T17:29:06.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.421+0000 7fcb78f9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "318e04fa-02d7-4358-879a-2613abedef73", "id": 3}] 2024-03-25T17:29:06.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.421+0000 7fcb78f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:06.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.421+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd new", "uuid": "318e04fa-02d7-4358-879a-2613abedef73", "id": 3} v 0) v1 -- 0x7fcb7413bb70 con 0x7fcb740b4700 2024-03-25T17:29:06.508 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.507+0000 7fcbb22c3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:06.510 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.510+0000 7f804442b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:06.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.540+0000 7fcb62ffd640 1 -- 172.21.15.203:0/3196653749 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "318e04fa-02d7-4358-879a-2613abedef73", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3684174990 0 525046306) 0x7fcb680056d0 con 0x7fcb740b4700 2024-03-25T17:29:06.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.540+0000 7fcb62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "318e04fa-02d7-4358-879a-2613abedef73", "id": 3}] 2024-03-25T17:29:06.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.540+0000 7fcb62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:06.540 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-03-25T17:29:06.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 10 librados: watch_flush enter 2024-03-25T17:29:06.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 10 librados: watch_flush exit 2024-03-25T17:29:06.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 20 client.4136.objecter shutdown clearing up homeless session... 2024-03-25T17:29:06.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 10 client.4136.objecter successfully canceled tick 2024-03-25T17:29:06.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 10 monclient: shutdown 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 >> v1:172.21.15.96:6790/0 conn(0x7fcb740b4700 legacy=0x7fcb741392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 shutdown_connections 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 >> 172.21.15.203:0/3196653749 conn(0x7fcb740668e0 msgr2=0x7fcb7413d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 shutdown_connections 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 -- 172.21.15.203:0/3196653749 wait complete. 2024-03-25T17:29:06.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.541+0000 7fcb78f9d640 1 librados: shutdown 2024-03-25T17:29:06.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 733daa39-6ad9-4192-b243-b61c78d9c006 4 2024-03-25T17:29:06.608 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.608+0000 7fcbb22c3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:29:06.612 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.612+0000 7f804442b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.651+0000 7f9d64fd6640 10 monclient: get_monmap_and_config 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.651+0000 7f9d64fd6640 10 monclient: build_initial_monmap 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.651+0000 7f9d64fd6640 10 monclient: monmap: 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:06.652416+0000 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:06.652416+0000 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:06.652 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.652+0000 7f9d64fd6640 1 Processor -- start 2024-03-25T17:29:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.652+0000 7f9d64fd6640 1 -- start start 2024-03-25T17:29:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.652+0000 7f9d64fd6640 10 monclient: init 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d600b4700 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d60142a20 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600a9d60 con 0x7f9d600b4700 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6013ff10 con 0x7f9d600a96b0 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600aa1f0 con 0x7f9d60142a20 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d64fd6640 10 monclient(hunting): authenticate will time out at 5296.960976s 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d5ed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9d600a96b0 0x7f9d6013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47136/0 (socket says 172.21.15.203:47136) 2024-03-25T17:29:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.653+0000 7f9d5ed9d640 1 -- 172.21.15.203:0/1314531131 learned_addr learned my addr 172.21.15.203:0/1314531131 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553864251 0 0) 0x7f9d600a9d60 con 0x7f9d600b4700 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): my global_id is 4140 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48003520 con 0x7f9d600b4700 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071485232 0 0) 0x7f9d6013ff10 con 0x7f9d600a96b0 2024-03-25T17:29:06.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): my global_id is 4135 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d600a9d60 con 0x7f9d600a96b0 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475578110 0 0) 0x7f9d600aa1f0 con 0x7f9d60142a20 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 10 monclient(hunting): my global_id is 4139 2024-03-25T17:29:06.658 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.656+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6013ff10 con 0x7f9d60142a20 2024-03-25T17:29:06.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982823748 0 0) 0x7f9d48003520 con 0x7f9d600b4700 2024-03-25T17:29:06.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 >> v1:172.21.15.96:6790/0 conn(0x7f9d60142a20 legacy=0x7f9d6013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 >> v1:172.21.15.203:6789/0 conn(0x7f9d600a96b0 legacy=0x7f9d6013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: _finish_hunting 0 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 monclient: found mon.noname-a 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600b4ae0 con 0x7f9d600b4700 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: _finish_auth 0 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: _check_auth_tickets 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d54001690 con 0x7f9d600b4700 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d64fd6640 5 monclient: authenticate success, global_id 4140 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d64fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d5d59a640 10 monclient: dump: 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.659+0000 7f9d64fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d540017f0 con 0x7f9d600b4700 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 1 -- 172.21.15.203:0/1314531131 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d54001c00 con 0x7f9d600b4700 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d64fd6640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d64fd6640 4 set_mon_vals no callback set 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:06.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d5d59a640 10 monclient: dump: 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.661+0000 7f9d64fd6640 10 monclient: shutdown 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 1 -- 172.21.15.203:0/1314531131 >> v1:172.21.15.96:6789/0 conn(0x7f9d600b4700 legacy=0x7f9d60142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 1 -- 172.21.15.203:0/1314531131 shutdown_connections 2024-03-25T17:29:06.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 1 -- 172.21.15.203:0/1314531131 >> 172.21.15.203:0/1314531131 conn(0x7f9d600668e0 msgr2=0x7f9d60066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:06.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 1 -- 172.21.15.203:0/1314531131 shutdown_connections 2024-03-25T17:29:06.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.662+0000 7f9d64fd6640 1 -- 172.21.15.203:0/1314531131 wait complete. 2024-03-25T17:29:06.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 10 monclient: build_initial_monmap 2024-03-25T17:29:06.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 10 monclient: monmap: 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 librados: starting msgr at 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 librados: starting objecter 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 Processor -- start 2024-03-25T17:29:06.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 -- start start 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 librados: setting wanted keys 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 1 librados: calling monclient init 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 10 monclient: init 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.663+0000 7f9d64fd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600a96b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d600b4700 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d60142a20 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6013a4d0 con 0x7f9d60142a20 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d601da880 con 0x7f9d600b4700 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d60139240 con 0x7f9d600a96b0 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d64fd6640 10 monclient(hunting): authenticate will time out at 5296.971701s 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d5dd9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9d600b4700 0x7f9d601db040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47138/0 (socket says 172.21.15.203:47138) 2024-03-25T17:29:06.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d5dd9b640 1 -- 172.21.15.203:0/192478634 learned_addr learned my addr 172.21.15.203:0/192478634 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d4effd640 10 client.?.objecter ms_handle_connect 0x7f9d600b4700 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d4effd640 10 client.?.objecter ms_handle_connect 0x7f9d60142a20 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.664+0000 7f9d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.665+0000 7f9d4effd640 10 client.?.objecter ms_handle_connect 0x7f9d600a96b0 2024-03-25T17:29:06.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.665+0000 7f9d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605274451 0 0) 0x7f9d6013a4d0 con 0x7f9d60142a20 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 10 monclient(hunting): my global_id is 4143 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.666+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d40004eb0 con 0x7f9d60142a20 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589374415 0 0) 0x7f9d601da880 con 0x7f9d600b4700 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): my global_id is 4138 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6013a4d0 con 0x7f9d600b4700 2024-03-25T17:29:06.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303359894 0 0) 0x7f9d60139240 con 0x7f9d600a96b0 2024-03-25T17:29:06.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:06.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:06.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:06.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 10 monclient(hunting): my global_id is 4142 2024-03-25T17:29:06.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.667+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d601da880 con 0x7f9d600a96b0 2024-03-25T17:29:06.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755400512 0 0) 0x7f9d6013a4d0 con 0x7f9d600b4700 2024-03-25T17:29:06.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d60139240 con 0x7f9d600b4700 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d500019f0 con 0x7f9d600b4700 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.672 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725432657 0 0) 0x7f9d40004eb0 con 0x7f9d60142a20 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d6013a4d0 con 0x7f9d60142a20 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d58001b50 con 0x7f9d60142a20 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.671+0000 7f9d4effd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.673 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414832016 0 0) 0x7f9d601da880 con 0x7f9d600a96b0 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d40004eb0 con 0x7f9d600a96b0 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d54001b00 con 0x7f9d600a96b0 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.672+0000 7f9d4effd640 10 monclient(hunting): dump: 2024-03-25T17:29:06.674 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.675 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.674+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3088230400 0 0) 0x7f9d60139240 con 0x7f9d600b4700 2024-03-25T17:29:06.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.674+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 >> v1:172.21.15.96:6790/0 conn(0x7f9d600a96b0 legacy=0x7f9d60138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 >> v1:172.21.15.96:6789/0 conn(0x7f9d60142a20 legacy=0x7f9d601e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 1 monclient: found mon.b 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d601d4fe0 con 0x7f9d600b4700 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 10 monclient: _finish_auth 0 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 5 monclient: authenticate success, global_id 4138 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _renew_subs 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d601dab20 con 0x7f9d600b4700 2024-03-25T17:29:06.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _renew_subs 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601db800 con 0x7f9d600b4700 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 1 librados: init done 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:06.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.675+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d601dad10 con 0x7f9d600b4700 2024-03-25T17:29:06.767 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.767+0000 7fcbb22c3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:06.771 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.772+0000 7f804442b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:06.809 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.809+0000 7fcbb22c3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:06.814 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.814+0000 7f804442b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d50003610 con 0x7f9d600b4700 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9d50003920 con 0x7f9d600b4700 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: dump: 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9d50003ed0 con 0x7f9d600b4700 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d5cd99640 4 set_mon_vals no callback set 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3765320861 0 0) 0x7f9d500066f0 con 0x7f9d600b4700 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 client.4138.objecter ms_dispatch 0x7f9d60000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9d50006ad0 con 0x7f9d600b4700 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.846+0000 7f9d4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:06.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.917+0000 7f9d64fd6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "733daa39-6ad9-4192-b243-b61c78d9c006", "id": 4}] 2024-03-25T17:29:06.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.917+0000 7f9d64fd6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "733daa39-6ad9-4192-b243-b61c78d9c006", "id": 4}] 2024-03-25T17:29:06.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.917+0000 7f9d64fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:06.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:06.917+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "733daa39-6ad9-4192-b243-b61c78d9c006", "id": 4} v 0) v1 -- 0x7f9d6004f730 con 0x7f9d600b4700 2024-03-25T17:29:06.964 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:06.964+0000 7fcbb22c3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:29:06.971 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:06.971+0000 7f804442b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:29:07.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.055+0000 7f9d4effd640 1 -- 172.21.15.203:0/192478634 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "733daa39-6ad9-4192-b243-b61c78d9c006", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2087090405 0 1697012583) 0x7f9d50004180 con 0x7f9d600b4700 2024-03-25T17:29:07.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.055+0000 7f9d4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "733daa39-6ad9-4192-b243-b61c78d9c006", "id": 4}] 2024-03-25T17:29:07.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.055+0000 7f9d4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:07.055 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 10 librados: watch_flush enter 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 10 librados: watch_flush exit 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 10 client.4138.objecter successfully canceled tick 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 10 monclient: shutdown 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 >> v1:172.21.15.203:6789/0 conn(0x7f9d600b4700 legacy=0x7f9d601db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 shutdown_connections 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 >> 172.21.15.203:0/192478634 conn(0x7f9d600668e0 msgr2=0x7f9d6013e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 shutdown_connections 2024-03-25T17:29:07.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 -- 172.21.15.203:0/192478634 wait complete. 2024-03-25T17:29:07.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.056+0000 7f9d64fd6640 1 librados: shutdown 2024-03-25T17:29:07.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new c5cc3897-7dcb-44b1-b07b-66d0ccc18405 5 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.167+0000 7f67bafe8640 10 monclient: get_monmap_and_config 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.167+0000 7f67bafe8640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.167+0000 7f67bafe8640 10 monclient: monmap: 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:07.167678+0000 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:07.167678+0000 2024-03-25T17:29:07.167 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.168 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.168 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.168 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:07.168 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:07.168 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.167+0000 7f67bafe8640 1 Processor -- start 2024-03-25T17:29:07.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.167+0000 7f67bafe8640 1 -- start start 2024-03-25T17:29:07.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient: init 2024-03-25T17:29:07.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-a con 0x7f67b40b4700 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-c con 0x7f67b4142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-b con 0x7f67b40a96b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40a9d60 con 0x7f67b40b4700 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b413ff10 con 0x7f67b4142a20 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40aa1f0 con 0x7f67b40a96b0 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.168+0000 7f67bafe8640 10 monclient(hunting): authenticate will time out at 5297.476263s 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.169+0000 7f67b3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f67b4142a20 0x7f67b413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47148/0 (socket says 172.21.15.203:47148) 2024-03-25T17:29:07.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.169+0000 7f67b3fff640 1 -- 172.21.15.203:0/3740577748 learned_addr learned my addr 172.21.15.203:0/3740577748 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460681756 0 0) 0x7f67b413ff10 con 0x7f67b4142a20 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a4003520 con 0x7f67b4142a20 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232506561 0 0) 0x7f67b40a9d60 con 0x7f67b40b4700 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.171+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b413ff10 con 0x7f67b40b4700 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681137725 0 0) 0x7f67b40aa1f0 con 0x7f67b40a96b0 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T17:29:07.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.172+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b40a9d60 con 0x7f67b40a96b0 2024-03-25T17:29:07.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142662446 0 0) 0x7f67a4003520 con 0x7f67b4142a20 2024-03-25T17:29:07.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 >> v1:172.21.15.96:6790/0 conn(0x7f67b40a96b0 legacy=0x7f67b413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 >> v1:172.21.15.96:6789/0 conn(0x7f67b40b4700 legacy=0x7f67b4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 monclient: found mon.noname-c 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b40b4ae0 con 0x7f67b4142a20 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67a0001690 con 0x7f67b4142a20 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67bafe8640 5 monclient: authenticate success, global_id 4141 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67bafe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67b37fe640 10 monclient: dump: 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.176 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.175+0000 7f67bafe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67a00017f0 con 0x7f67b4142a20 2024-03-25T17:29:07.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 4 set_mon_vals no callback set 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 1 -- 172.21.15.203:0/3740577748 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67a0001c00 con 0x7f67b4142a20 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67b37fe640 10 monclient: dump: 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 10 monclient: shutdown 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 1 -- 172.21.15.203:0/3740577748 >> v1:172.21.15.203:6789/0 conn(0x7f67b4142a20 legacy=0x7f67b413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 1 -- 172.21.15.203:0/3740577748 shutdown_connections 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 1 -- 172.21.15.203:0/3740577748 >> 172.21.15.203:0/3740577748 conn(0x7f67b40668e0 msgr2=0x7f67b4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 1 -- 172.21.15.203:0/3740577748 shutdown_connections 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.177+0000 7f67bafe8640 1 -- 172.21.15.203:0/3740577748 wait complete. 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.178+0000 7f67bafe8640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.178+0000 7f67bafe8640 10 monclient: monmap: 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.182 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.178+0000 7f67bafe8640 1 librados: starting msgr at 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.178+0000 7f67bafe8640 1 librados: starting objecter 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 Processor -- start 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 -- start start 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 librados: setting wanted keys 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 librados: calling monclient init 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient: init 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-a con 0x7f67b40a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-b con 0x7f67b40b4700 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient(hunting): picked mon.noname-c con 0x7f67b4142a20 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b41365c0 con 0x7f67b40a96b0 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40ac120 con 0x7f67b40b4700 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40ac400 con 0x7f67b4142a20 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.179+0000 7f67bafe8640 10 monclient(hunting): authenticate will time out at 5297.487506s 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f67b40b4700 0x7f67b40ac820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47150/0 (socket says 172.21.15.203:47150) 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b3fff640 1 -- 172.21.15.203:0/2494201897 learned_addr learned my addr 172.21.15.203:0/2494201897 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter ms_handle_connect 0x7f67b40b4700 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter ms_handle_connect 0x7f67b40a96b0 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter ms_handle_connect 0x7f67b4142a20 2024-03-25T17:29:07.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.180+0000 7f67b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990571938 0 0) 0x7f67b40ac120 con 0x7f67b40b4700 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 10 monclient(hunting): my global_id is 4144 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.182+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798004eb0 con 0x7f67b40b4700 2024-03-25T17:29:07.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969896715 0 0) 0x7f67b41365c0 con 0x7f67b40a96b0 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): my global_id is 4149 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b40ac120 con 0x7f67b40a96b0 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573574486 0 0) 0x7f67b40ac400 con 0x7f67b4142a20 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 10 monclient(hunting): my global_id is 4148 2024-03-25T17:29:07.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.183+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b41365c0 con 0x7f67b4142a20 2024-03-25T17:29:07.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2235422288 0 0) 0x7f6798004eb0 con 0x7f67b40b4700 2024-03-25T17:29:07.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b40ac400 con 0x7f67b40b4700 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67a0001bc0 con 0x7f67b40b4700 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 10 monclient(hunting): dump: 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.187+0000 7f67b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:07.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947502160 0 0) 0x7f67b40ac120 con 0x7f67b40a96b0 2024-03-25T17:29:07.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6798004eb0 con 0x7f67b40a96b0 2024-03-25T17:29:07.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67a8001970 con 0x7f67b40a96b0 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): dump: 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899466969 0 0) 0x7f67b41365c0 con 0x7f67b4142a20 2024-03-25T17:29:07.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b40ac120 con 0x7f67b4142a20 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67ac001c70 con 0x7f67b4142a20 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.189+0000 7f67b17fa640 10 monclient(hunting): dump: 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.191 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2666948018 0 0) 0x7f67b40ac400 con 0x7f67b40b4700 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 >> v1:172.21.15.96:6790/0 conn(0x7f67b4142a20 legacy=0x7f67b41e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 >> v1:172.21.15.96:6789/0 conn(0x7f67b40a96b0 legacy=0x7f67b4135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 1 monclient: found mon.b 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b41350d0 con 0x7f67b40b4700 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 5 monclient: authenticate success, global_id 4144 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 10 monclient: _renew_subs 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b4136250 con 0x7f67b40b4700 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 10 monclient: _renew_subs 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b4133920 con 0x7f67b40b4700 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.191+0000 7f67bafe8640 1 librados: init done 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.192+0000 7f67bafe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.192+0000 7f67bafe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.192+0000 7f67bafe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.192+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b4133700 con 0x7f67b40b4700 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.290+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67a00011f0 con 0x7f67b40b4700 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.290+0000 7f67b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.290+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f67a0004c30 con 0x7f67b40b4700 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.290+0000 7f67b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 10 monclient: dump: 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.291 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f67a0005160 con 0x7f67b40b4700 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b2ffd640 4 set_mon_vals no callback set 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 556003834 0 0) 0x7f67a0006900 con 0x7f67b40b4700 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 10 client.4144.objecter ms_dispatch 0x7f67b4000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f67a0018e00 con 0x7f67b40b4700 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.291+0000 7f67b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.361+0000 7f67bafe8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c5cc3897-7dcb-44b1-b07b-66d0ccc18405", "id": 5}] 2024-03-25T17:29:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.362+0000 7f67bafe8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c5cc3897-7dcb-44b1-b07b-66d0ccc18405", "id": 5}] 2024-03-25T17:29:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.362+0000 7f67bafe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.362+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c5cc3897-7dcb-44b1-b07b-66d0ccc18405", "id": 5} v 0) v1 -- 0x7f67b40b1d70 con 0x7f67b40b4700 2024-03-25T17:29:07.422 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.422+0000 7f67b17fa640 1 -- 172.21.15.203:0/2494201897 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5cc3897-7dcb-44b1-b07b-66d0ccc18405", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2179965630 0 1988412176) 0x7f67a00063f0 con 0x7f67b40b4700 2024-03-25T17:29:07.422 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.422+0000 7f67b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c5cc3897-7dcb-44b1-b07b-66d0ccc18405", "id": 5}] 2024-03-25T17:29:07.422 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.422+0000 7f67b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:07.422 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-03-25T17:29:07.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 10 librados: watch_flush enter 2024-03-25T17:29:07.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 10 librados: watch_flush exit 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 10 client.4144.objecter successfully canceled tick 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 10 monclient: shutdown 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 >> v1:172.21.15.203:6789/0 conn(0x7f67b40b4700 legacy=0x7f67b40ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 shutdown_connections 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 >> 172.21.15.203:0/2494201897 conn(0x7f67b40668e0 msgr2=0x7f67b413e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 shutdown_connections 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 -- 172.21.15.203:0/2494201897 wait complete. 2024-03-25T17:29:07.424 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.423+0000 7f67bafe8640 1 librados: shutdown 2024-03-25T17:29:07.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 154501bc-27c0-44aa-b485-1c14c8037882 6 2024-03-25T17:29:07.533 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.533+0000 7f9154af6640 10 monclient: get_monmap_and_config 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.533+0000 7f9154af6640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.533+0000 7f9154af6640 10 monclient: monmap: 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:07.534344+0000 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:07.534344+0000 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:07.534 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.534+0000 7f9154af6640 1 Processor -- start 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.534+0000 7f9154af6640 1 -- start start 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.534+0000 7f9154af6640 10 monclient: init 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-c con 0x7f91500b4700 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-b con 0x7f9150142a20 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-a con 0x7f91500a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500a9d60 con 0x7f91500a96b0 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915013ff10 con 0x7f91500b4700 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500aa1f0 con 0x7f9150142a20 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.535+0000 7f9154af6640 10 monclient(hunting): authenticate will time out at 5297.843033s 2024-03-25T17:29:07.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.536+0000 7f914e59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f91500b4700 0x7f9150142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47164/0 (socket says 172.21.15.203:47164) 2024-03-25T17:29:07.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.536+0000 7f914e59c640 1 -- 172.21.15.203:0/3193761730 learned_addr learned my addr 172.21.15.203:0/3193761730 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628344763 0 0) 0x7f915013ff10 con 0x7f91500b4700 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 10 monclient(hunting): my global_id is 4147 2024-03-25T17:29:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.538+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913c003520 con 0x7f91500b4700 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251213384 0 0) 0x7f91500a9d60 con 0x7f91500a96b0 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 10 monclient(hunting): my global_id is 4152 2024-03-25T17:29:07.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.540+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915013ff10 con 0x7f91500a96b0 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435510784 0 0) 0x7f91500aa1f0 con 0x7f9150142a20 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 10 monclient(hunting): my global_id is 4151 2024-03-25T17:29:07.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.541+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91500a9d60 con 0x7f9150142a20 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295200811 0 0) 0x7f915013ff10 con 0x7f91500a96b0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 >> v1:172.21.15.96:6790/0 conn(0x7f9150142a20 legacy=0x7f915013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 >> v1:172.21.15.203:6789/0 conn(0x7f91500b4700 legacy=0x7f9150142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 monclient: found mon.noname-a 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91500b4ae0 con 0x7f91500a96b0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f91400019f0 con 0x7f91500a96b0 2024-03-25T17:29:07.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f9154af6640 5 monclient: authenticate success, global_id 4152 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f9154af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f914d59a640 10 monclient: dump: 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.543+0000 7f9154af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9140001b50 con 0x7f91500a96b0 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 1 -- 172.21.15.203:0/3193761730 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9140001fa0 con 0x7f91500a96b0 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f9154af6640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f9154af6640 4 set_mon_vals no callback set 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f914d59a640 10 monclient: dump: 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.548 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f9154af6640 10 monclient: shutdown 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f9154af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.545+0000 7f9154af6640 1 -- 172.21.15.203:0/3193761730 >> v1:172.21.15.96:6789/0 conn(0x7f91500a96b0 legacy=0x7f915013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 1 -- 172.21.15.203:0/3193761730 shutdown_connections 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 1 -- 172.21.15.203:0/3193761730 >> 172.21.15.203:0/3193761730 conn(0x7f91500668e0 msgr2=0x7f9150066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 1 -- 172.21.15.203:0/3193761730 shutdown_connections 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 1 -- 172.21.15.203:0/3193761730 wait complete. 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.546+0000 7f9154af6640 10 monclient: monmap: 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 librados: starting msgr at 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 librados: starting objecter 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 Processor -- start 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 -- start start 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 librados: setting wanted keys 2024-03-25T17:29:07.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 librados: calling monclient init 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient: init 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-a con 0x7f91500a96b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-b con 0x7f91500b4700 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient(hunting): picked mon.noname-c con 0x7f9150142a20 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150137620 con 0x7f91500a96b0 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150137890 con 0x7f91500b4700 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150136800 con 0x7f9150142a20 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.547+0000 7f9154af6640 10 monclient(hunting): authenticate will time out at 5297.855590s 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f914dd9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f91500b4700 0x7f9150132a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47176/0 (socket says 172.21.15.203:47176) 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f914dd9b640 1 -- 172.21.15.203:0/3697063743 learned_addr learned my addr 172.21.15.203:0/3697063743 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f9146ffd640 10 client.?.objecter ms_handle_connect 0x7f91500b4700 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f9146ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f9146ffd640 10 client.?.objecter ms_handle_connect 0x7f9150142a20 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.548+0000 7f9146ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.549+0000 7f9146ffd640 10 client.?.objecter ms_handle_connect 0x7f91500a96b0 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.549+0000 7f9146ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039470911 0 0) 0x7f9150136800 con 0x7f9150142a20 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130004eb0 con 0x7f9150142a20 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485839989 0 0) 0x7f9150137890 con 0x7f91500b4700 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.550+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150136800 con 0x7f91500b4700 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396605864 0 0) 0x7f9150137620 con 0x7f91500a96b0 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T17:29:07.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.551+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150137890 con 0x7f91500a96b0 2024-03-25T17:29:07.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110475054 0 0) 0x7f9150137890 con 0x7f91500a96b0 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9150137620 con 0x7f91500a96b0 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9148001230 con 0x7f91500a96b0 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.557 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.555+0000 7f9146ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45322704 0 0) 0x7f9150136800 con 0x7f91500b4700 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9150137890 con 0x7f91500b4700 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9138001950 con 0x7f91500b4700 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.558 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3674955027 0 0) 0x7f9130004eb0 con 0x7f9150142a20 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9150136800 con 0x7f9150142a20 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9140001ea0 con 0x7f9150142a20 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.556+0000 7f9146ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.559 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.560 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.560 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.560 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.560 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.560 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469723970 0 0) 0x7f9150137890 con 0x7f91500b4700 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 >> v1:172.21.15.96:6790/0 conn(0x7f9150142a20 legacy=0x7f91501e26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 >> v1:172.21.15.96:6789/0 conn(0x7f91500a96b0 legacy=0x7f9150135290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 1 monclient: found mon.b 2024-03-25T17:29:07.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150135ce0 con 0x7f91500b4700 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 5 monclient: authenticate success, global_id 4150 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 10 monclient: _renew_subs 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9150135ed0 con 0x7f91500b4700 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 10 monclient: _renew_subs 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9150133330 con 0x7f91500b4700 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.560+0000 7f9154af6640 1 librados: init done 2024-03-25T17:29:07.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.561+0000 7f9154af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.561+0000 7f9154af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.561+0000 7f9154af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.561+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91500ab1c0 con 0x7f91500b4700 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.564+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9138003610 con 0x7f91500b4700 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9138003920 con 0x7f91500b4700 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 10 monclient: dump: 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.565 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f914cd99640 4 set_mon_vals no callback set 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9138003ed0 con 0x7f91500b4700 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1831620671 0 0) 0x7f9138006750 con 0x7f91500b4700 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 10 client.4150.objecter ms_dispatch 0x7f9150000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T17:29:07.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.565+0000 7f9146ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T17:29:07.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.659+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9138004180 con 0x7f91500b4700 2024-03-25T17:29:07.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.659+0000 7f9146ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.659+0000 7f9146ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:07.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.731+0000 7f9154af6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "154501bc-27c0-44aa-b485-1c14c8037882", "id": 6}] 2024-03-25T17:29:07.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.731+0000 7f9154af6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "154501bc-27c0-44aa-b485-1c14c8037882", "id": 6}] 2024-03-25T17:29:07.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.731+0000 7f9154af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.731+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "154501bc-27c0-44aa-b485-1c14c8037882", "id": 6} v 0) v1 -- 0x7f9150136110 con 0x7f91500b4700 2024-03-25T17:29:07.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.790+0000 7f9146ffd640 1 -- 172.21.15.203:0/3697063743 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "154501bc-27c0-44aa-b485-1c14c8037882", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3850096350 0 1113811849) 0x7f9138004450 con 0x7f91500b4700 2024-03-25T17:29:07.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.790+0000 7f9146ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "154501bc-27c0-44aa-b485-1c14c8037882", "id": 6}] 2024-03-25T17:29:07.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.790+0000 7f9146ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:07.790 INFO:teuthology.orchestra.run.smithi203.stdout:6 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 10 librados: watch_flush enter 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 10 librados: watch_flush exit 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 10 client.4150.objecter successfully canceled tick 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 10 monclient: shutdown 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 >> v1:172.21.15.203:6789/0 conn(0x7f91500b4700 legacy=0x7f9150132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 shutdown_connections 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 >> 172.21.15.203:0/3697063743 conn(0x7f91500668e0 msgr2=0x7f915013e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 shutdown_connections 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 -- 172.21.15.203:0/3697063743 wait complete. 2024-03-25T17:29:07.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.791+0000 7f9154af6640 1 librados: shutdown 2024-03-25T17:29:07.797 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:07.797+0000 7fcbb22c3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:29:07.801 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:07.800+0000 7f804442b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:29:07.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 76585eac-e095-4e42-acf6-482d6dbad102 7 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 10 monclient: get_monmap_and_config 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 10 monclient: monmap: 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:29:07.902748+0000 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:29:07.902748+0000 2024-03-25T17:29:07.902 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.903 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.903 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.903 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:07.903 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:07.903 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 1 Processor -- start 2024-03-25T17:29:07.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 1 -- start start 2024-03-25T17:29:07.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.902+0000 7f0ca164e640 10 monclient: init 2024-03-25T17:29:07.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c9c0b4700 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c9c142a20 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-b con 0x7f0c9c0a96b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c0a9d60 con 0x7f0c9c0b4700 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c13ff10 con 0x7f0c9c142a20 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c0aa1f0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.903+0000 7f0ca164e640 10 monclient(hunting): authenticate will time out at 5298.211239s 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.904+0000 7f0c9affd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0c9c0b4700 0x7f0c9c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.203:58476/0 (socket says 172.21.15.203:58476) 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.904+0000 7f0c9affd640 1 -- 172.21.15.203:0/2025346404 learned_addr learned my addr 172.21.15.203:0/2025346404 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.904+0000 7f0c9a7fc640 1 --1- 172.21.15.203:0/2025346404 >> v1:172.21.15.203:6789/0 conn(0x7f0c9c142a20 0x7f0c9c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47182/0 (socket says 172.21.15.203:47182) 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620687862 0 0) 0x7f0c9c13ff10 con 0x7f0c9c142a20 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 10 monclient(hunting): my global_id is 4153 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.906+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c84003520 con 0x7f0c9c142a20 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342355963 0 0) 0x7f0c9c0aa1f0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): my global_id is 4157 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c13ff10 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113038124 0 0) 0x7f0c9c0a9d60 con 0x7f0c9c0b4700 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 10 monclient(hunting): my global_id is 4158 2024-03-25T17:29:07.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.907+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c0aa1f0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2102255497 0 0) 0x7f0c84003520 con 0x7f0c9c142a20 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 >> v1:172.21.15.96:6790/0 conn(0x7f0c9c0a96b0 legacy=0x7f0c9c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 >> v1:172.21.15.96:6789/0 conn(0x7f0c9c0b4700 legacy=0x7f0c9c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 monclient: found mon.noname-c 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c0b4ae0 con 0x7f0c9c142a20 2024-03-25T17:29:07.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0c8c001690 con 0x7f0c9c142a20 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0ca164e640 5 monclient: authenticate success, global_id 4153 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0ca164e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0c99ffb640 10 monclient: dump: 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.911 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.912 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.912 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.912 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.909+0000 7f0ca164e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c8c0017f0 con 0x7f0c9c142a20 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 1 -- 172.21.15.203:0/2025346404 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0c8c001c00 con 0x7f0c9c142a20 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0ca164e640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0ca164e640 4 set_mon_vals no callback set 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.911+0000 7f0c99ffb640 10 monclient: dump: 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 10 monclient: shutdown 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 1 -- 172.21.15.203:0/2025346404 >> v1:172.21.15.203:6789/0 conn(0x7f0c9c142a20 legacy=0x7f0c9c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 1 -- 172.21.15.203:0/2025346404 shutdown_connections 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 1 -- 172.21.15.203:0/2025346404 >> 172.21.15.203:0/2025346404 conn(0x7f0c9c0668e0 msgr2=0x7f0c9c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 1 -- 172.21.15.203:0/2025346404 shutdown_connections 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.912+0000 7f0ca164e640 1 -- 172.21.15.203:0/2025346404 wait complete. 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 10 monclient: build_initial_monmap 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 10 monclient: monmap: 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:07.916 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 librados: starting msgr at 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 librados: starting objecter 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 Processor -- start 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 -- start start 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 librados: setting wanted keys 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 1 librados: calling monclient init 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.913+0000 7f0ca164e640 10 monclient: init 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-b con 0x7f0c9c0a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c9c0b4700 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c9c142a20 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c14eca0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c1d74d0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c1d8350 con 0x7f0c9c142a20 2024-03-25T17:29:07.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:07.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0ca164e640 10 monclient(hunting): authenticate will time out at 5298.221957s 2024-03-25T17:29:07.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0c9affd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0c9c0a96b0 0x7f0c9c1d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:47188/0 (socket says 172.21.15.203:47188) 2024-03-25T17:29:07.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.914+0000 7f0c9affd640 1 -- 172.21.15.203:0/3998044784 learned_addr learned my addr 172.21.15.203:0/3998044784 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter ms_handle_connect 0x7f0c9c0a96b0 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter ms_handle_connect 0x7f0c9c142a20 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter ms_handle_connect 0x7f0c9c0b4700 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.915+0000 7f0c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807495753 0 0) 0x7f0c9c1d74d0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T17:29:07.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.916+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c88004eb0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600226341 0 0) 0x7f0c9c0b4ae0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.918+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c1d74d0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536161433 0 0) 0x7f0c9c1d8350 con 0x7f0c9c142a20 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T17:29:07.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.919+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c0b4ae0 con 0x7f0c9c142a20 2024-03-25T17:29:07.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042695595 0 0) 0x7f0c88004eb0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c9c1d8350 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0c90001910 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 10 monclient(hunting): dump: 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.921+0000 7f0c837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688801365 0 0) 0x7f0c9c1d74d0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c88004eb0 con 0x7f0c9c0b4700 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0c8c001960 con 0x7f0c9c0b4700 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient(hunting): dump: 2024-03-25T17:29:07.925 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1471611868 0 0) 0x7f0c9c1d8350 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 >> v1:172.21.15.96:6790/0 conn(0x7f0c9c142a20 legacy=0x7f0c9c1d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 >> v1:172.21.15.96:6789/0 conn(0x7f0c9c0b4700 legacy=0x7f0c9c0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 monclient: found mon.b 2024-03-25T17:29:07.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c0abd90 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0c837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 5 monclient: authenticate success, global_id 4156 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 10 monclient: _renew_subs 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c1d7720 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 10 monclient: _renew_subs 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c0ad240 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.927 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.924+0000 7f0ca164e640 1 librados: init done 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.926+0000 7f0ca164e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.926+0000 7f0ca164e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.926+0000 7f0ca164e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.926+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c04f730 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c900047e0 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0c90004b70 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 10 monclient: dump: 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:07.929 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c997fa640 4 set_mon_vals no callback set 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0c90005120 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2365101159 0 0) 0x7f0c90006840 con 0x7f0c9c0a96b0 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 10 client.4156.objecter ms_dispatch 0x7f0c9c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T17:29:07.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:07.928+0000 7f0c837fe640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T17:29:08.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.026+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0c900053d0 con 0x7f0c9c0a96b0 2024-03-25T17:29:08.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.026+0000 7f0c837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:08.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.026+0000 7f0c837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:08.085 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:08.084+0000 7fcbb22c3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:29:08.088 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:08.088+0000 7f804442b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:29:08.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.104+0000 7f0ca164e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "76585eac-e095-4e42-acf6-482d6dbad102", "id": 7}] 2024-03-25T17:29:08.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.104+0000 7f0ca164e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "76585eac-e095-4e42-acf6-482d6dbad102", "id": 7}] 2024-03-25T17:29:08.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.104+0000 7f0ca164e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:29:08.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.104+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "76585eac-e095-4e42-acf6-482d6dbad102", "id": 7} v 0) v1 -- 0x7f0c9c0b7470 con 0x7f0c9c0a96b0 2024-03-25T17:29:08.140 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:08.140+0000 7f804442b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:29:08.141 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:08.140+0000 7fcbb22c3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:29:08.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.162+0000 7f0c837fe640 1 -- 172.21.15.203:0/3998044784 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76585eac-e095-4e42-acf6-482d6dbad102", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3447265066 0 1371665406) 0x7f0c90005570 con 0x7f0c9c0a96b0 2024-03-25T17:29:08.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.162+0000 7f0c837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "76585eac-e095-4e42-acf6-482d6dbad102", "id": 7}] 2024-03-25T17:29:08.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.162+0000 7f0c837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:08.163 INFO:teuthology.orchestra.run.smithi203.stdout:7 2024-03-25T17:29:08.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 10 librados: watch_flush enter 2024-03-25T17:29:08.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 10 librados: watch_flush exit 2024-03-25T17:29:08.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T17:29:08.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 10 client.4156.objecter successfully canceled tick 2024-03-25T17:29:08.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 10 monclient: shutdown 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 >> v1:172.21.15.203:6789/0 conn(0x7f0c9c0a96b0 legacy=0x7f0c9c1d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 shutdown_connections 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.163+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 >> 172.21.15.203:0/3998044784 conn(0x7f0c9c0668e0 msgr2=0x7f0c9c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.164+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 shutdown_connections 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.164+0000 7f0ca164e640 1 -- 172.21.15.203:0/3998044784 wait complete. 2024-03-25T17:29:08.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T17:29:08.164+0000 7f0ca164e640 1 librados: shutdown 2024-03-25T17:29:08.178 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.178 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T17:29:08.178 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T17:29:08.180 INFO:tasks.ceph.osd.0:Started 2024-03-25T17:29:08.180 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.181 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T17:29:08.181 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T17:29:08.183 INFO:tasks.ceph.osd.1:Started 2024-03-25T17:29:08.183 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.183 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T17:29:08.183 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T17:29:08.185 INFO:tasks.ceph.osd.2:Started 2024-03-25T17:29:08.185 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.185 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T17:29:08.185 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T17:29:08.187 INFO:tasks.ceph.osd.3:Started 2024-03-25T17:29:08.187 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.188 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T17:29:08.188 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T17:29:08.190 INFO:tasks.ceph.osd.4:Started 2024-03-25T17:29:08.190 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.190 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T17:29:08.190 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T17:29:08.192 INFO:tasks.ceph.osd.5:Started 2024-03-25T17:29:08.192 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.192 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T17:29:08.192 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T17:29:08.194 INFO:tasks.ceph.osd.6:Started 2024-03-25T17:29:08.194 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:08.195 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T17:29:08.195 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T17:29:08.197 INFO:tasks.ceph.osd.7:Started 2024-03-25T17:29:08.197 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:08.276 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:08.276+0000 7f804442b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:29:08.297 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-03-25T17:29:08.296+0000 7fcbb22c3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.319+0000 7fb07c0d3640 10 monclient: get_monmap_and_config 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.319+0000 7fb07c0d3640 10 monclient: build_initial_monmap 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.319+0000 7fb07c0d3640 10 monclient: monmap: 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 0 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:29:08.320535+0000 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:29:08.320535+0000 2024-03-25T17:29:08.320 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.319+0000 7fb07c0d3640 1 Processor -- start 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 1 -- start start 2024-03-25T17:29:08.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient: init 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb0740b1d40 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb0740b14e0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb0740b4400 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07413ff90 con 0x7fb0740b14e0 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740a9a60 con 0x7fb0740b4400 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740ab4a0 con 0x7fb0740b1d40 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.320+0000 7fb07c0d3640 10 monclient(hunting): authenticate will time out at 5306.602630s 2024-03-25T17:29:08.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.321+0000 7fb07966e640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb0740b14e0 0x7fb074142340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54942/0 (socket says 172.21.15.96:54942) 2024-03-25T17:29:08.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.321+0000 7fb079e6f640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fb0740b1d40 0x7fb0740b0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:47132/0 (socket says 172.21.15.96:47132) 2024-03-25T17:29:08.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.321+0000 7fb07966e640 1 -- 172.21.15.96:0/1307281415 learned_addr learned my addr 172.21.15.96:0/1307281415 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105770093 0 0) 0x7fb07413ff90 con 0x7fb0740b14e0 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 10 monclient(hunting): _negotiate 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 10 monclient(hunting): my global_id is 4164 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb060003520 con 0x7fb0740b14e0 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.323+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166242597 0 0) 0x7fb0740ab4a0 con 0x7fb0740b1d40 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.324+0000 7fb078e6d640 10 monclient(hunting): _negotiate 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.324+0000 7fb078e6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.324+0000 7fb078e6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.324+0000 7fb078e6d640 10 monclient(hunting): my global_id is 4163 2024-03-25T17:29:08.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.324+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07413ff90 con 0x7fb0740b1d40 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 437074333 0 0) 0x7fb07413ff90 con 0x7fb0740b1d40 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 >> v1:172.21.15.203:6789/0 conn(0x7fb0740b4400 legacy=0x7fb07414dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 >> v1:172.21.15.96:6789/0 conn(0x7fb0740b14e0 legacy=0x7fb074142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: _finish_hunting 0 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 monclient: found mon.noname-b 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740a9540 con 0x7fb0740b1d40 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: _finish_auth 0 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: _check_auth_tickets 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:08.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb070001670 con 0x7fb0740b1d40 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb078e6d640 10 monclient: dump: 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb07c0d3640 5 monclient: authenticate success, global_id 4163 2024-03-25T17:29:08.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.327+0000 7fb07c0d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.329+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0700017d0 con 0x7fb0740b1d40 2024-03-25T17:29:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb078e6d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb078e6d640 1 -- 172.21.15.96:0/1307281415 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb070001be0 con 0x7fb0740b1d40 2024-03-25T17:29:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb078e6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb078e6d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb078e6d640 10 monclient: dump: 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 4 set_mon_vals no callback set 2024-03-25T17:29:08.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 10 monclient: shutdown 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 1 -- 172.21.15.96:0/1307281415 >> v1:172.21.15.96:6790/0 conn(0x7fb0740b1d40 legacy=0x7fb0740b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 1 -- 172.21.15.96:0/1307281415 shutdown_connections 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.330+0000 7fb07c0d3640 1 -- 172.21.15.96:0/1307281415 >> 172.21.15.96:0/1307281415 conn(0x7fb0740b7df0 msgr2=0x7fb0740b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.331+0000 7fb07c0d3640 1 -- 172.21.15.96:0/1307281415 shutdown_connections 2024-03-25T17:29:08.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.331+0000 7fb07c0d3640 1 -- 172.21.15.96:0/1307281415 wait complete. 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.331+0000 7fb07c0d3640 10 monclient: build_initial_monmap 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 10 monclient: monmap: 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 0 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:created 0.000000 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:08.335 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 librados: starting msgr at 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 librados: starting objecter 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 Processor -- start 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 -- start start 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 librados: setting wanted keys 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 1 librados: calling monclient init 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.332+0000 7fb07c0d3640 10 monclient: init 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb0740b14e0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb0740b1d40 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:08.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb0740b4400 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb074072130 con 0x7fb0740b14e0 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb074070a80 con 0x7fb0740b4400 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb074070cd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.333+0000 7fb07c0d3640 10 monclient(hunting): authenticate will time out at 5306.615448s 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.334+0000 7fb07966e640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fb0740b1d40 0x7fb07406f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:47136/0 (socket says 172.21.15.96:47136) 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.334+0000 7fb07966e640 1 -- 172.21.15.96:0/959010605 learned_addr learned my addr 172.21.15.96:0/959010605 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.334+0000 7fb06a7fc640 10 client.?.objecter ms_handle_connect 0x7fb0740b1d40 2024-03-25T17:29:08.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.334+0000 7fb06a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125076002 0 0) 0x7fb074070cd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 10 monclient(hunting): my global_id is 4166 2024-03-25T17:29:08.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.337+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb058004eb0 con 0x7fb0740b1d40 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396097807 0 0) 0x7fb058004eb0 con 0x7fb0740b1d40 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb074070cd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb064001a70 con 0x7fb0740b1d40 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:08.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 10 monclient(hunting): dump: 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.343+0000 7fb06a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.348+0000 7fb06a7fc640 10 client.?.objecter ms_handle_connect 0x7fb0740b4400 2024-03-25T17:29:08.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.348+0000 7fb06a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 368501787 0 0) 0x7fb074070cd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 >> v1:172.21.15.203:6789/0 conn(0x7fb0740b4400 legacy=0x7fb0741e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 >> v1:172.21.15.96:6789/0 conn(0x7fb0740b14e0 legacy=0x7fb074071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 1 monclient: found mon.c 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074070fb0 con 0x7fb0740b1d40 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb06a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 5 monclient: authenticate success, global_id 4166 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 10 monclient: _renew_subs 2024-03-25T17:29:08.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07406fdd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 10 monclient: _renew_subs 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0741cbfe0 con 0x7fb0740b1d40 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.350+0000 7fb07c0d3640 1 librados: init done 2024-03-25T17:29:08.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.351+0000 7fb07c0d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.351+0000 7fb07c0d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.351+0000 7fb07c0d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.351+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0741cc2e0 con 0x7fb0740b1d40 2024-03-25T17:29:08.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb064001d10 con 0x7fb0740b1d40 2024-03-25T17:29:08.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:08.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb064004c30 con 0x7fb0740b1d40 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 10 monclient: dump: 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:epoch 1 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb0640051e0 con 0x7fb0740b1d40 2024-03-25T17:29:08.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fb064006b30 con 0x7fb0740b1d40 2024-03-25T17:29:08.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 10 client.4166.objecter ms_dispatch 0x7fb074000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:08.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:08.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.394+0000 7fb06a7fc640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:08.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.395+0000 7fb06a7fc640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T17:29:08.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.395+0000 7fb06bfff640 4 set_mon_vals no callback set 2024-03-25T17:29:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.467+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb064006570 con 0x7fb0740b1d40 2024-03-25T17:29:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.467+0000 7fb06a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.467+0000 7fb06a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.558+0000 7fb07c0d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.558+0000 7fb07c0d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.558+0000 7fb07c0d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.558+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0741cbcd0 con 0x7fb0740b1d40 2024-03-25T17:29:08.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.608+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3554422044 0 0) 0x7fb0640067e0 con 0x7fb0740b1d40 2024-03-25T17:29:08.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.749+0000 7fb06a7fc640 1 -- 172.21.15.96:0/959010605 <== mon.2 v1:172.21.15.96:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fb064016250 con 0x7fb0740b1d40 2024-03-25T17:29:08.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.749+0000 7fb06a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:08.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.749+0000 7fb06a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-25T17:29:08.751 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 10 librados: watch_flush enter 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 10 librados: watch_flush exit 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 10 client.4166.objecter successfully canceled tick 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 10 monclient: shutdown 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 >> v1:172.21.15.96:6790/0 conn(0x7fb0740b1d40 legacy=0x7fb07406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.750+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 shutdown_connections 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.751+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 >> 172.21.15.96:0/959010605 conn(0x7fb0740b7df0 msgr2=0x7fb0740b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.751+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 shutdown_connections 2024-03-25T17:29:08.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.751+0000 7fb07c0d3640 1 -- 172.21.15.96:0/959010605 wait complete. 2024-03-25T17:29:08.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-25T17:29:08.751+0000 7fb07c0d3640 1 librados: shutdown 2024-03-25T17:29:08.767 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T17:29:08.767 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T17:29:09.573 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:09.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.723+0000 7f2e5a902640 10 monclient: get_monmap_and_config 2024-03-25T17:29:09.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.723+0000 7f2e5a902640 10 monclient: build_initial_monmap 2024-03-25T17:29:09.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.723+0000 7f2e5a902640 10 monclient: monmap: 2024-03-25T17:29:09.724 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:09.724 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:09.724918+0000 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:09.724918+0000 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:09.725 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.724+0000 7f2e5a902640 1 Processor -- start 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.724+0000 7f2e5a902640 1 -- start start 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.724+0000 7f2e5a902640 10 monclient: init 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-a con 0x7f2e540b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-b con 0x7f2e540b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e540af940 con 0x7f2e540b4d60 2024-03-25T17:29:09.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e540b0a70 con 0x7f2e5414be90 2024-03-25T17:29:09.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e540b0d50 con 0x7f2e540b1730 2024-03-25T17:29:09.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:09.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.725+0000 7f2e5a902640 10 monclient(hunting): authenticate will time out at 5308.007920s 2024-03-25T17:29:09.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.726+0000 7f2e537fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f2e540b1730 0x7f2e540b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:47144/0 (socket says 172.21.15.96:47144) 2024-03-25T17:29:09.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:09.726+0000 7f2e537fe640 1 -- 172.21.15.96:0/709257741 learned_addr learned my addr 172.21.15.96:0/709257741 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:10.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211469453 0 0) 0x7f2e540af940 con 0x7f2e540b4d60 2024-03-25T17:29:10.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:10.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:10.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:10.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T17:29:10.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.045+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e3c003520 con 0x7f2e540b4d60 2024-03-25T17:29:10.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335678059 0 0) 0x7f2e3c003520 con 0x7f2e540b4d60 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 >> v1:172.21.15.96:6790/0 conn(0x7f2e540b1730 legacy=0x7f2e540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 >> v1:172.21.15.203:6789/0 conn(0x7f2e5414be90 legacy=0x7f2e5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 1 monclient: found mon.noname-a 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e540b1030 con 0x7f2e540b4d60 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:10.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e5a902640 5 monclient: authenticate success, global_id 4176 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.050+0000 7f2e5a902640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.051+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e44001690 con 0x7f2e540b4d60 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.051+0000 7f2e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.051+0000 7f2e52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.051+0000 7f2e52ffd640 10 monclient: dump: 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:10.052 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:10.053 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.051+0000 7f2e5a902640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:10.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e440018e0 con 0x7f2e540b4d60 2024-03-25T17:29:10.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 1 -- 172.21.15.96:0/709257741 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e44001c70 con 0x7f2e540b4d60 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e52ffd640 10 monclient: dump: 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:10.055 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.052+0000 7f2e5a902640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.053+0000 7f2e5a902640 4 set_mon_vals no callback set 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.053+0000 7f2e5a902640 10 monclient: shutdown 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.053+0000 7f2e5a902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.053+0000 7f2e5a902640 1 -- 172.21.15.96:0/709257741 >> v1:172.21.15.96:6789/0 conn(0x7f2e540b4d60 legacy=0x7f2e5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.054+0000 7f2e5a902640 1 -- 172.21.15.96:0/709257741 shutdown_connections 2024-03-25T17:29:10.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.054+0000 7f2e5a902640 1 -- 172.21.15.96:0/709257741 >> 172.21.15.96:0/709257741 conn(0x7f2e540617a0 msgr2=0x7f2e54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:10.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.054+0000 7f2e5a902640 1 -- 172.21.15.96:0/709257741 shutdown_connections 2024-03-25T17:29:10.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.054+0000 7f2e5a902640 1 -- 172.21.15.96:0/709257741 wait complete. 2024-03-25T17:29:10.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.055+0000 7f2e5a902640 10 monclient: build_initial_monmap 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.055+0000 7f2e5a902640 10 monclient: monmap: 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:10.058 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:10.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.055+0000 7f2e5a902640 1 librados: starting msgr at 2024-03-25T17:29:10.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.055+0000 7f2e5a902640 1 librados: starting objecter 2024-03-25T17:29:10.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.056+0000 7f2e5a902640 1 Processor -- start 2024-03-25T17:29:10.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.057+0000 7f2e5a902640 1 -- start start 2024-03-25T17:29:10.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.057+0000 7f2e5a902640 1 librados: setting wanted keys 2024-03-25T17:29:10.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.057+0000 7f2e5a902640 1 librados: calling monclient init 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.057+0000 7f2e5a902640 10 monclient: init 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.058+0000 7f2e5a902640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.058+0000 7f2e5a902640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.058+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-a con 0x7f2e540b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.058+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-c con 0x7f2e540b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.058+0000 7f2e5a902640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e5a902640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e541386f0 con 0x7f2e540b1730 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e5a902640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e541378c0 con 0x7f2e5414be90 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e5a902640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e54138940 con 0x7f2e540b4d60 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e53fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e540b1730 0x7f2e54136550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54970/0 (socket says 172.21.15.96:54970) 2024-03-25T17:29:10.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e53fff640 1 -- 172.21.15.96:0/706848370 learned_addr learned my addr 172.21.15.96:0/706848370 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:10.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e5a902640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:10.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e5a902640 10 monclient(hunting): authenticate will time out at 5308.341364s 2024-03-25T17:29:10.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e50ff9640 10 client.?.objecter ms_handle_connect 0x7f2e540b1730 2024-03-25T17:29:10.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.059+0000 7f2e50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882151929 0 0) 0x7f2e541386f0 con 0x7f2e540b1730 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 10 monclient(hunting): my global_id is 4179 2024-03-25T17:29:10.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.063+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e40004eb0 con 0x7f2e540b1730 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1104278468 0 0) 0x7f2e40004eb0 con 0x7f2e540b1730 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e541386f0 con 0x7f2e540b1730 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e44001cb0 con 0x7f2e540b1730 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:10.072 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:10.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.070+0000 7f2e50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:10.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557184973 0 0) 0x7f2e541386f0 con 0x7f2e540b1730 2024-03-25T17:29:10.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 >> v1:172.21.15.96:6790/0 conn(0x7f2e540b4d60 legacy=0x7f2e54142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 >> v1:172.21.15.203:6789/0 conn(0x7f2e5414be90 legacy=0x7f2e541d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 1 monclient: found mon.a 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e54143270 con 0x7f2e540b1730 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e5a902640 5 monclient: authenticate success, global_id 4179 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e5a902640 10 monclient: _renew_subs 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e5a902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e54137270 con 0x7f2e540b1730 2024-03-25T17:29:10.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.076+0000 7f2e5a902640 10 client.4179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.077+0000 7f2e5a902640 10 monclient: _renew_subs 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.077+0000 7f2e5a902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.077+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e541d8360 con 0x7f2e540b1730 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.077+0000 7f2e5a902640 1 librados: init done 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.078+0000 7f2e5a902640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.078+0000 7f2e5a902640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.078+0000 7f2e5a902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.078+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e540627b0 con 0x7f2e540b1730 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e44003960 con 0x7f2e540b1730 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e44003d70 con 0x7f2e540b1730 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e50ff9640 10 monclient: dump: 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:10.113 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.112+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 4245778657 0 0) 0x7f2e44003f90 con 0x7f2e540b1730 2024-03-25T17:29:10.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.111+0000 7f2e527fc640 4 set_mon_vals no callback set 2024-03-25T17:29:10.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.114+0000 7f2e537fe640 10 monclient: get_auth_request con 0x7f2e4006da40 auth_method 0 2024-03-25T17:29:10.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.114+0000 7f2e50ff9640 10 client.4179.objecter ms_handle_connect 0x7f2e4006da40 2024-03-25T17:29:10.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.117+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f2e44054a80 con 0x7f2e540b1730 2024-03-25T17:29:10.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.117+0000 7f2e50ff9640 10 client.4179.objecter ms_dispatch 0x7f2e54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:10.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.117+0000 7f2e50ff9640 3 client.4179.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:10.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.117+0000 7f2e50ff9640 3 client.4179.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:10.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.117+0000 7f2e50ff9640 20 client.4179.objecter dump_active .. 0 homeless 2024-03-25T17:29:10.309 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:10.308+0000 7f7fe4991640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T17:29:10.313 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T17:29:10.313+0000 7f7fe4991640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T17:29:10.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.536+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e44050590 con 0x7f2e540b1730 2024-03-25T17:29:10.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.536+0000 7f2e50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:10.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.536+0000 7f2e50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:10.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.870+0000 7f2e5a902640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:10.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.870+0000 7f2e5a902640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:10.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.870+0000 7f2e5a902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:10.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.870+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e54137400 con 0x7f2e540b1730 2024-03-25T17:29:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.941+0000 7f2e50ff9640 1 -- 172.21.15.96:0/706848370 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f2e440524c0 con 0x7f2e540b1730 2024-03-25T17:29:10.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.941+0000 7f2e50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:10.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.941+0000 7f2e50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:10.942 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:10.942 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:10.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 10 librados: watch_flush enter 2024-03-25T17:29:10.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 10 librados: watch_flush exit 2024-03-25T17:29:10.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 20 client.4179.objecter shutdown clearing up homeless session... 2024-03-25T17:29:10.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 10 client.4179.objecter successfully canceled tick 2024-03-25T17:29:10.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 >> v1:172.21.15.203:6800/3611964061 conn(0x7f2e4006da40 legacy=0x7f2e4006ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 10 monclient: shutdown 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 >> v1:172.21.15.96:6789/0 conn(0x7f2e540b1730 legacy=0x7f2e54136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 shutdown_connections 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 >> 172.21.15.96:0/706848370 conn(0x7f2e540617a0 msgr2=0x7f2e540b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 shutdown_connections 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 -- 172.21.15.96:0/706848370 wait complete. 2024-03-25T17:29:10.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:10.944+0000 7f2e5a902640 1 librados: shutdown 2024-03-25T17:29:10.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:15.127 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:06.848 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.162 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:06.907 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.179 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:06.919 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.188 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:06.909 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.226 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:06.971 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.241 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:06.980 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.386 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:07.135 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.454 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:07.182 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.468 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:07.216 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.504 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:07.225 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.515 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:07.244 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.565 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:07.286 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.706 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:07.443 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.737 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-25T17:29:15.729+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:15.768 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:07.506 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.788 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-25T17:29:15.781+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:15.804 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-25T17:29:15.796+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:15.841 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:07.589 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:15.900 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:07.649 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.002 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-03-25T17:29:15.997+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:16.085 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-03-25T17:29:16.079+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:16.123 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-25T17:29:16.115+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:16.231 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:07.952 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.267 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.011 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.267 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.006 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.334 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-03-25T17:29:16.326+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:16.474 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:08.222 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.500 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-03-25T17:29:16.494+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:16.575 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:08.304 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.644 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.365 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.645 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.390 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.646 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.367 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.668 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.408 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.802 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:08.540 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.852 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:08.600 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.957 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:08.685 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.967 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.687 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:16.986 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:08.735 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.000 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.739 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.013 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.734 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.018 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.763 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.143 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:08.880 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.174 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:08.923 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.231 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.952 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.265 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.005 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.268 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:08.997 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.323 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.068 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.352 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.072 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.357 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:09.105 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.439 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:09.187 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.460 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:09.197 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.495 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.216 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.530 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.270 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.533 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:09.261 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.612 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.356 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.616 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.337 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.645 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:09.394 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.703 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:09.451 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.730 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:09.467 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.763 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.484 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.765 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:17.800 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:09.528 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.803 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.543 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 10 monclient: get_monmap_and_config 2024-03-25T17:29:17.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 10 monclient: build_initial_monmap 2024-03-25T17:29:17.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 10 monclient: monmap: 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:17.854434+0000 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:17.854434+0000 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:17.855 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:17.856 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 1 Processor -- start 2024-03-25T17:29:17.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 1 -- start start 2024-03-25T17:29:17.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.854+0000 7ff7fa53b640 10 monclient: init 2024-03-25T17:29:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f406ea40 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f4071800 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f4071c40 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f406f1f0 con 0x7ff7f4071c40 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4070d00 con 0x7ff7f406ea40 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f40720b0 con 0x7ff7f4071800 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7fa53b640 10 monclient(hunting): authenticate will time out at 5316.136477s 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7f8ad8640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff7f4071c40 0x7ff7f414a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58308/0 (socket says 172.21.15.96:58308) 2024-03-25T17:29:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.855+0000 7ff7f8ad8640 1 -- 172.21.15.96:0/1996414188 learned_addr learned my addr 172.21.15.96:0/1996414188 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:17.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641231086 0 0) 0x7ff7f406f1f0 con 0x7ff7f4071c40 2024-03-25T17:29:17.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003440 con 0x7ff7f4071c40 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723131740 0 0) 0x7ff7f4070d00 con 0x7ff7f406ea40 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): my global_id is 4198 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f406f1f0 con 0x7ff7f406ea40 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250769836 0 0) 0x7ff7f40720b0 con 0x7ff7f4071800 2024-03-25T17:29:17.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 10 monclient(hunting): my global_id is 4205 2024-03-25T17:29:17.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.858+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f4070d00 con 0x7ff7f4071800 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3235990506 0 0) 0x7ff7f406f1f0 con 0x7ff7f406ea40 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 >> v1:172.21.15.96:6790/0 conn(0x7ff7f4071800 legacy=0x7ff7f4147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 >> v1:172.21.15.96:6789/0 conn(0x7ff7f4071c40 legacy=0x7ff7f414a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 monclient: found mon.noname-c 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:17.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f4072780 con 0x7ff7f406ea40 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7fa53b640 5 monclient: authenticate success, global_id 4198 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7fa53b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7e4001630 con 0x7ff7f406ea40 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:17.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7f2ffd640 10 monclient: dump: 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.861+0000 7ff7fa53b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7e4001880 con 0x7ff7f406ea40 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 1 -- 172.21.15.96:0/1996414188 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7e4001c10 con 0x7ff7f406ea40 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:17.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7f2ffd640 10 monclient: dump: 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7fa53b640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.863+0000 7ff7fa53b640 4 set_mon_vals no callback set 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 10 monclient: shutdown 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 -- 172.21.15.96:0/1996414188 >> v1:172.21.15.203:6789/0 conn(0x7ff7f406ea40 legacy=0x7ff7f413ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 -- 172.21.15.96:0/1996414188 shutdown_connections 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 -- 172.21.15.96:0/1996414188 >> 172.21.15.96:0/1996414188 conn(0x7ff7f406b630 msgr2=0x7ff7f40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 -- 172.21.15.96:0/1996414188 shutdown_connections 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 -- 172.21.15.96:0/1996414188 wait complete. 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 10 monclient: build_initial_monmap 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 10 monclient: monmap: 2024-03-25T17:29:17.871 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 librados: starting msgr at 2024-03-25T17:29:17.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.864+0000 7ff7fa53b640 1 librados: starting objecter 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 Processor -- start 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 -- start start 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 librados: setting wanted keys 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 librados: calling monclient init 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient: init 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f406ea40 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f4071800 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f4071c40 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:17.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f41db1b0 con 0x7ff7f4071c40 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f411c6f0 con 0x7ff7f4071800 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4138ec0 con 0x7ff7f406ea40 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.865+0000 7ff7fa53b640 10 monclient(hunting): authenticate will time out at 5316.146767s 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f8ad8640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff7f4071c40 0x7ff7f41387b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58322/0 (socket says 172.21.15.96:58322) 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f8ad8640 1 -- 172.21.15.96:0/911383065 learned_addr learned my addr 172.21.15.96:0/911383065 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f4071c40 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f4071800 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f406ea40 2024-03-25T17:29:17.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.866+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694893080 0 0) 0x7ff7f411c6f0 con 0x7ff7f4071800 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4201 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d4004eb0 con 0x7ff7f4071800 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569270163 0 0) 0x7ff7f41db1b0 con 0x7ff7f4071c40 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4209 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f411c6f0 con 0x7ff7f4071c40 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973928696 0 0) 0x7ff7f4138ec0 con 0x7ff7f406ea40 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4208 2024-03-25T17:29:17.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.868+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f41db1b0 con 0x7ff7f406ea40 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3518729940 0 0) 0x7ff7d4004eb0 con 0x7ff7f4071800 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f4138ec0 con 0x7ff7f4071800 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7dc0018f0 con 0x7ff7f4071800 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:17.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371040351 0 0) 0x7ff7f411c6f0 con 0x7ff7f4071c40 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.872+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7d4004eb0 con 0x7ff7f4071c40 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263201759 0 0) 0x7ff7f41db1b0 con 0x7ff7f406ea40 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f411c6f0 con 0x7ff7f406ea40 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7e4001b90 con 0x7ff7f406ea40 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.873+0000 7ff7f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.881 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7ec001b10 con 0x7ff7f4071c40 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:17.882 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 810033275 0 0) 0x7ff7d4004eb0 con 0x7ff7f4071c40 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 >> v1:172.21.15.96:6790/0 conn(0x7ff7f406ea40 legacy=0x7ff7f41d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 >> v1:172.21.15.203:6789/0 conn(0x7ff7f4071800 legacy=0x7ff7f41380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:17.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 monclient: found mon.a 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f41dac00 con 0x7ff7f4071c40 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 5 monclient: authenticate success, global_id 4209 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 10 monclient: _renew_subs 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f41daed0 con 0x7ff7f4071c40 2024-03-25T17:29:17.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 10 monclient: _renew_subs 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f41c91e0 con 0x7ff7f4071c40 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.876+0000 7ff7fa53b640 1 librados: init done 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.877+0000 7ff7fa53b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.877+0000 7ff7fa53b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.877+0000 7ff7fa53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:17.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:17.877+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f40627b0 con 0x7ff7f4071c40 2024-03-25T17:29:17.886 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.601 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.903 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.647 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.919 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:09.667 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:17.971 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:09.719 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.000 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:09.737 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.031 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.752 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.067 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.807 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.068 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:09.796 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7ec001fe0 con 0x7ff7f4071c40 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff7ec004d00 con 0x7ff7f4071c40 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f0ff9640 10 monclient: dump: 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:18.117 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.116+0000 7ff7f27fc640 4 set_mon_vals no callback set 2024-03-25T17:29:18.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.117+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ff7ec01acd0 con 0x7ff7f4071c40 2024-03-25T17:29:18.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.118+0000 7ff7f3fff640 10 monclient: get_auth_request con 0x7ff7d406e0a0 auth_method 0 2024-03-25T17:29:18.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.118+0000 7ff7f0ff9640 10 client.4209.objecter ms_handle_connect 0x7ff7d406e0a0 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ff7ec054a70 con 0x7ff7f4071c40 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 10 client.4209.objecter ms_dispatch 0x7ff7f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 20 client.4209.objecter dump_active .. 0 homeless 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7ec0503b0 con 0x7ff7f4071c40 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:18.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.122+0000 7ff7f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:18.144 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.865 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.186 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:09.934 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.191 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.935 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.239 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:09.987 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.271 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:10.008 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.295 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.016 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.296+0000 7ff7fa53b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:18.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.296+0000 7ff7fa53b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:18.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.296+0000 7ff7fa53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:18.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.296+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7f40b0110 con 0x7ff7f4071c40 2024-03-25T17:29:18.312 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.312+0000 7ff7f0ff9640 1 -- 172.21.15.96:0/911383065 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ff7ec0524b0 con 0x7ff7f4071c40 2024-03-25T17:29:18.312 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.312+0000 7ff7f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:18.312 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.312+0000 7ff7f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:18.312 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:18.312 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.313+0000 7ff7fa53b640 10 librados: watch_flush enter 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 10 librados: watch_flush exit 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 20 client.4209.objecter shutdown clearing up homeless session... 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 10 client.4209.objecter successfully canceled tick 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 >> v1:172.21.15.203:6800/3611964061 conn(0x7ff7d406e0a0 legacy=0x7ff7d4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 10 monclient: shutdown 2024-03-25T17:29:18.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 >> v1:172.21.15.96:6789/0 conn(0x7ff7f4071c40 legacy=0x7ff7f41387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 shutdown_connections 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 >> 172.21.15.96:0/911383065 conn(0x7ff7f406b630 msgr2=0x7ff7f40b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 shutdown_connections 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 -- 172.21.15.96:0/911383065 wait complete. 2024-03-25T17:29:18.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:18.314+0000 7ff7fa53b640 1 librados: shutdown 2024-03-25T17:29:18.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:18.333 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.073 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.339 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:10.067 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.412 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.133 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.463 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:10.211 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.478 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.222 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.503 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:10.251 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.536 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:10.273 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.562 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.280 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.597 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.337 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.603 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:10.331 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.680 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.401 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.732 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:10.480 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.765 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.509 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.770 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:10.518 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.809 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:10.546 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.871 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:10.599 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.954 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.675 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:18.997 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:10.745 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.040 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.761 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.073 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:10.810 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.091 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.831 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.094 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.815 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.146 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.885 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.249 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:10.997 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.269 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:11.017 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.286 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.030 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.304 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:11.052 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.341 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.085 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.358 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:11.086 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.421 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:11.149 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.472 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:11.193 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.530 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:11.250 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.566 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:11.303 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.625 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:11.362 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.752 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:11.500 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:19.806 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:11.554 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.235 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:15.956 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.324 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:16.064 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.470 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:16.215 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.556 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:16.304 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.616 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:16.344 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.693 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:16.414 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:24.955 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:16.703 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:25.129 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:25.214 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:16.951 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.247+0000 7f2e72158640 10 monclient: get_monmap_and_config 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.247+0000 7f2e72158640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.247+0000 7f2e72158640 10 monclient: monmap: 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:25.249074+0000 2024-03-25T17:29:25.248 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:25.249074+0000 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:25.249 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.248+0000 7f2e72158640 1 Processor -- start 2024-03-25T17:29:25.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.248+0000 7f2e72158640 1 -- start start 2024-03-25T17:29:25.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.248+0000 7f2e72158640 10 monclient: init 2024-03-25T17:29:25.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-b con 0x7f2e6c0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6c0b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-c con 0x7f2e6c14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0af940 con 0x7f2e6c0b1730 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0b0a70 con 0x7f2e6c14be90 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0b0d50 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.249+0000 7f2e72158640 10 monclient(hunting): authenticate will time out at 5323.531399s 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.250+0000 7f2e6affd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e6c0b1730 0x7f2e6c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38884/0 (socket says 172.21.15.96:38884) 2024-03-25T17:29:25.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.250+0000 7f2e6affd640 1 -- 172.21.15.96:0/1924712181 learned_addr learned my addr 172.21.15.96:0/1924712181 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201141313 0 0) 0x7f2e6c0b0d50 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54003520 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049947607 0 0) 0x7f2e6c0af940 con 0x7f2e6c0b1730 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.252+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c0b0d50 con 0x7f2e6c0b1730 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631341932 0 0) 0x7f2e6c0b0a70 con 0x7f2e6c14be90 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 10 monclient(hunting): my global_id is 4204 2024-03-25T17:29:25.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.253+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c0af940 con 0x7f2e6c14be90 2024-03-25T17:29:25.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635559265 0 0) 0x7f2e54003520 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 >> v1:172.21.15.203:6789/0 conn(0x7f2e6c14be90 legacy=0x7f2e6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 >> v1:172.21.15.96:6789/0 conn(0x7f2e6c0b1730 legacy=0x7f2e6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 monclient: found mon.noname-b 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0b1030 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e72158640 5 monclient: authenticate success, global_id 4211 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e72158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e5c001690 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.256+0000 7f2e6a7fc640 10 monclient: dump: 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.257+0000 7f2e72158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.259+0000 7f2e6a7fc640 1 -- 172.21.15.96:0/1924712181 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e5c001870 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.259+0000 7f2e6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.259+0000 7f2e72158640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.259+0000 7f2e72158640 4 set_mon_vals no callback set 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 10 monclient: shutdown 2024-03-25T17:29:25.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 1 -- 172.21.15.96:0/1924712181 >> v1:172.21.15.96:6790/0 conn(0x7f2e6c0b4d60 legacy=0x7f2e6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 1 -- 172.21.15.96:0/1924712181 shutdown_connections 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 1 -- 172.21.15.96:0/1924712181 >> 172.21.15.96:0/1924712181 conn(0x7f2e6c0617a0 msgr2=0x7f2e6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 1 -- 172.21.15.96:0/1924712181 shutdown_connections 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.260+0000 7f2e72158640 1 -- 172.21.15.96:0/1924712181 wait complete. 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 10 monclient: monmap: 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:25.265 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 librados: starting msgr at 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 librados: starting objecter 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 Processor -- start 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 -- start start 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 librados: setting wanted keys 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 1 librados: calling monclient init 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.261+0000 7f2e72158640 10 monclient: init 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:25.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-b con 0x7f2e6c0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-c con 0x7f2e6c0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6c14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c139da0 con 0x7f2e6c14be90 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c138ed0 con 0x7f2e6c0b1730 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c139060 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e72158640 10 monclient(hunting): authenticate will time out at 5323.544451s 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e6bfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e6c14be90 0x7f2e6c1428f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38896/0 (socket says 172.21.15.96:38896) 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.262+0000 7f2e6bfff640 1 -- 172.21.15.96:0/2937699337 learned_addr learned my addr 172.21.15.96:0/2937699337 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter ms_handle_connect 0x7f2e6c14be90 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter ms_handle_connect 0x7f2e6c0b1730 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter ms_handle_connect 0x7f2e6c0b4d60 2024-03-25T17:29:25.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.263+0000 7f2e53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139870004 0 0) 0x7f2e6c139da0 con 0x7f2e6c14be90 2024-03-25T17:29:25.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e58004eb0 con 0x7f2e6c14be90 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582919680 0 0) 0x7f2e6c139060 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.265+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c139da0 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187921920 0 0) 0x7f2e6c138ed0 con 0x7f2e6c0b1730 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 10 monclient(hunting): my global_id is 4207 2024-03-25T17:29:25.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.266+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c139060 con 0x7f2e6c0b1730 2024-03-25T17:29:25.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889207814 0 0) 0x7f2e58004eb0 con 0x7f2e6c14be90 2024-03-25T17:29:25.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6c138ed0 con 0x7f2e6c14be90 2024-03-25T17:29:25.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e64001c70 con 0x7f2e6c14be90 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 10 monclient(hunting): dump: 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.272+0000 7f2e53fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 285660217 0 0) 0x7f2e6c139da0 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e58004eb0 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091188280 0 0) 0x7f2e6c139060 con 0x7f2e6c0b1730 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6c139da0 con 0x7f2e6c0b1730 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e60001930 con 0x7f2e6c0b4d60 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): dump: 2024-03-25T17:29:25.275 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e5c001690 con 0x7f2e6c0b1730 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.273+0000 7f2e53fff640 10 monclient(hunting): dump: 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:25.276 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:25.277 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4098615183 0 0) 0x7f2e6c138ed0 con 0x7f2e6c14be90 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 >> v1:172.21.15.96:6790/0 conn(0x7f2e6c0b4d60 legacy=0x7f2e6c1421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 >> v1:172.21.15.203:6789/0 conn(0x7f2e6c0b1730 legacy=0x7f2e6c140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 1 monclient: found mon.a 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c1451c0 con 0x7f2e6c14be90 2024-03-25T17:29:25.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 5 monclient: authenticate success, global_id 4215 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 10 monclient: _renew_subs 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c145470 con 0x7f2e6c14be90 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 10 monclient: _renew_subs 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.276+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c1e7970 con 0x7f2e6c14be90 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.277+0000 7f2e72158640 1 librados: init done 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.278+0000 7f2e72158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.278+0000 7f2e72158640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.278+0000 7f2e72158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:25.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.278+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c0afbd0 con 0x7f2e6c14be90 2024-03-25T17:29:25.533 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e64003990 con 0x7f2e6c14be90 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2e64003da0 con 0x7f2e6c14be90 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e53fff640 10 monclient: dump: 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:25.534 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.533+0000 7f2e69ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.534+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f2e64003fc0 con 0x7f2e6c14be90 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.534+0000 7f2e6b7fe640 10 monclient: get_auth_request con 0x7f2e5806e190 auth_method 0 2024-03-25T17:29:25.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.535+0000 7f2e53fff640 10 client.4215.objecter ms_handle_connect 0x7f2e5806e190 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f2e64054960 con 0x7f2e6c14be90 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 10 client.4215.objecter ms_dispatch 0x7f2e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e64050470 con 0x7f2e6c14be90 2024-03-25T17:29:25.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.541+0000 7f2e53fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:25.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.814+0000 7f2e72158640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:25.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.814+0000 7f2e72158640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:25.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.814+0000 7f2e72158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:25.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.814+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e6c1411f0 con 0x7f2e6c14be90 2024-03-25T17:29:25.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.829+0000 7f2e53fff640 1 -- 172.21.15.96:0/2937699337 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f2e640523a0 con 0x7f2e6c14be90 2024-03-25T17:29:25.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.829+0000 7f2e53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:25.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.829+0000 7f2e53fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:25.830 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:25.830 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 10 librados: watch_flush enter 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 10 librados: watch_flush exit 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 10 client.4215.objecter successfully canceled tick 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 >> v1:172.21.15.203:6800/3611964061 conn(0x7f2e5806e190 legacy=0x7f2e58070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 10 monclient: shutdown 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 >> v1:172.21.15.96:6789/0 conn(0x7f2e6c14be90 legacy=0x7f2e6c1428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 shutdown_connections 2024-03-25T17:29:25.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.831+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 >> 172.21.15.96:0/2937699337 conn(0x7f2e6c0617a0 msgr2=0x7f2e6c0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.832+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 shutdown_connections 2024-03-25T17:29:25.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.832+0000 7f2e72158640 1 -- 172.21.15.96:0/2937699337 wait complete. 2024-03-25T17:29:25.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:25.832+0000 7f2e72158640 1 librados: shutdown 2024-03-25T17:29:25.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:30.067 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:21.787 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.070 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:21.791 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.285 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:22.006 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.289 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:22.010 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.314 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:22.062 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.318 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:22.066 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.412 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:22.140 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.417 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:22.145 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.422 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.167 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.428 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.173 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.597 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:22.345 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.601 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:22.349 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.659 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:22.399 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.663 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:22.403 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.879 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:22.616 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:30.885 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:22.622 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:32.648 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.737+0000 7f56267b5640 10 monclient: get_monmap_and_config 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.737+0000 7f56267b5640 10 monclient: build_initial_monmap 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.738+0000 7f56267b5640 10 monclient: monmap: 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:32.739248+0000 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:32.739248+0000 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:32.739 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:32.740 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:32.740 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 1 Processor -- start 2024-03-25T17:29:32.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 1 -- start start 2024-03-25T17:29:32.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 10 monclient: init 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-b con 0x7f56200a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.739+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-a con 0x7f562013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-c con 0x7f56201430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620142520 con 0x7f562013f710 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200b28a0 con 0x7f56201430c0 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200b2a30 con 0x7f56200a98f0 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f56267b5640 10 monclient(hunting): authenticate will time out at 5331.022105s 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f5624d52640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f56201430c0 0x7f562014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:58694/0 (socket says 172.21.15.96:58694) 2024-03-25T17:29:32.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.740+0000 7f5624d52640 1 -- 172.21.15.96:0/2999775092 learned_addr learned my addr 172.21.15.96:0/2999775092 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:32.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096089794 0 0) 0x7f56200b28a0 con 0x7f56201430c0 2024-03-25T17:29:32.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 10 monclient(hunting): my global_id is 4210 2024-03-25T17:29:32.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.743+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5608003520 con 0x7f56201430c0 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304911043 0 0) 0x7f5620142520 con 0x7f562013f710 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 10 monclient(hunting): my global_id is 4218 2024-03-25T17:29:32.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.744+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56200b28a0 con 0x7f562013f710 2024-03-25T17:29:32.749 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144107052 0 0) 0x7f56200b2a30 con 0x7f56200a98f0 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient(hunting): my global_id is 4217 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5620142520 con 0x7f56200a98f0 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675139538 0 0) 0x7f5608003520 con 0x7f56201430c0 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 >> v1:172.21.15.96:6790/0 conn(0x7f56200a98f0 legacy=0x7f56200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 >> v1:172.21.15.96:6789/0 conn(0x7f562013f710 legacy=0x7f56201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 monclient: found mon.noname-c 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200b44c0 con 0x7f56201430c0 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: _finish_auth 0 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5618001a10 con 0x7f56201430c0 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f561effd640 10 monclient: dump: 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:32.751 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:32.752 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:32.752 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f56267b5640 5 monclient: authenticate success, global_id 4210 2024-03-25T17:29:32.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.747+0000 7f56267b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5618001b70 con 0x7f56201430c0 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 1 -- 172.21.15.96:0/2999775092 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5618001f80 con 0x7f56201430c0 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:32.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f561effd640 10 monclient: dump: 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f56267b5640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.750+0000 7f56267b5640 4 set_mon_vals no callback set 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 10 monclient: shutdown 2024-03-25T17:29:32.755 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 1 -- 172.21.15.96:0/2999775092 >> v1:172.21.15.203:6789/0 conn(0x7f56201430c0 legacy=0x7f562014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 1 -- 172.21.15.96:0/2999775092 shutdown_connections 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 1 -- 172.21.15.96:0/2999775092 >> 172.21.15.96:0/2999775092 conn(0x7f56200b7ea0 msgr2=0x7f56200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 1 -- 172.21.15.96:0/2999775092 shutdown_connections 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.751+0000 7f56267b5640 1 -- 172.21.15.96:0/2999775092 wait complete. 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.752+0000 7f56267b5640 10 monclient: build_initial_monmap 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.752+0000 7f56267b5640 10 monclient: monmap: 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:32.756 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.752+0000 7f56267b5640 1 librados: starting msgr at 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.752+0000 7f56267b5640 1 librados: starting objecter 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.753+0000 7f56267b5640 1 Processor -- start 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.753+0000 7f56267b5640 1 -- start start 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.753+0000 7f56267b5640 1 librados: setting wanted keys 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.753+0000 7f56267b5640 1 librados: calling monclient init 2024-03-25T17:29:32.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.753+0000 7f56267b5640 10 monclient: init 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-a con 0x7f56200a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-b con 0x7f562013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient(hunting): picked mon.noname-c con 0x7f56201430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620136ec0 con 0x7f56200a98f0 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620136c20 con 0x7f562013f710 2024-03-25T17:29:32.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620137db0 con 0x7f56201430c0 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f56267b5640 10 monclient(hunting): authenticate will time out at 5331.036591s 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f561ffff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f56200a98f0 0x7f56201376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38912/0 (socket says 172.21.15.96:38912) 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.754+0000 7f561ffff640 1 -- 172.21.15.96:0/4039146499 learned_addr learned my addr 172.21.15.96:0/4039146499 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.755+0000 7f561cff9640 10 client.?.objecter ms_handle_connect 0x7f562013f710 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.755+0000 7f561cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.755+0000 7f561cff9640 10 client.?.objecter ms_handle_connect 0x7f56200a98f0 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.755+0000 7f561cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 client.?.objecter ms_handle_connect 0x7f56201430c0 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152424035 0 0) 0x7f5620136ec0 con 0x7f56200a98f0 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 10 monclient(hunting): my global_id is 4221 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.757+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c004eb0 con 0x7f56200a98f0 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020215870 0 0) 0x7f5620136c20 con 0x7f562013f710 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 10 monclient(hunting): my global_id is 4213 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.758+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5620136ec0 con 0x7f562013f710 2024-03-25T17:29:32.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117222264 0 0) 0x7f5620137db0 con 0x7f56201430c0 2024-03-25T17:29:32.762 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:32.762 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:32.762 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:32.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 10 monclient(hunting): my global_id is 4220 2024-03-25T17:29:32.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.761+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5620136c20 con 0x7f56201430c0 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2495014092 0 0) 0x7f5620136ec0 con 0x7f562013f710 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5620137db0 con 0x7f562013f710 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5614001970 con 0x7f562013f710 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): dump: 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.766 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929992078 0 0) 0x7f560c004eb0 con 0x7f56200a98f0 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5620136ec0 con 0x7f56200a98f0 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5610001a10 con 0x7f56200a98f0 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.764+0000 7f561cff9640 10 monclient(hunting): dump: 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.767 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:32.768 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2331934197 0 0) 0x7f5620136ec0 con 0x7f56200a98f0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 >> v1:172.21.15.96:6790/0 conn(0x7f56201430c0 legacy=0x7f56201e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 >> v1:172.21.15.203:6789/0 conn(0x7f562013f710 legacy=0x7f56200b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 1 monclient: found mon.a 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5620138b50 con 0x7f56200a98f0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f561cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 5 monclient: authenticate success, global_id 4221 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 10 monclient: _renew_subs 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:32.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5620139830 con 0x7f56200a98f0 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 10 monclient: _renew_subs 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56201369b0 con 0x7f56200a98f0 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.768+0000 7f56267b5640 1 librados: init done 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.769+0000 7f56267b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.769+0000 7f56267b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.769+0000 7f56267b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:32.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.769+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5620139a20 con 0x7f56200a98f0 2024-03-25T17:29:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5610001c70 con 0x7f56200a98f0 2024-03-25T17:29:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f56100038c0 con 0x7f56200a98f0 2024-03-25T17:29:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561cff9640 10 monclient: dump: 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:32.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.827+0000 7f561e7fc640 4 set_mon_vals no callback set 2024-03-25T17:29:32.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.829+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5610003ae0 con 0x7f56200a98f0 2024-03-25T17:29:32.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.829+0000 7f561f7fe640 10 monclient: get_auth_request con 0x7f560c06dfb0 auth_method 0 2024-03-25T17:29:32.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.830+0000 7f561cff9640 10 client.4221.objecter ms_handle_connect 0x7f560c06dfb0 2024-03-25T17:29:32.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.916+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5610054710 con 0x7f56200a98f0 2024-03-25T17:29:32.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.916+0000 7f561cff9640 10 client.4221.objecter ms_dispatch 0x7f5620000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:32.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.916+0000 7f561cff9640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:32.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.916+0000 7f561cff9640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:32.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:32.916+0000 7f561cff9640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T17:29:33.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.025+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5610050220 con 0x7f56200a98f0 2024-03-25T17:29:33.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.025+0000 7f561cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:33.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.025+0000 7f561cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:33.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.314+0000 7f56267b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:33.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.314+0000 7f56267b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:33.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.314+0000 7f56267b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:33.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.314+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56200627b0 con 0x7f56200a98f0 2024-03-25T17:29:33.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.329+0000 7f561cff9640 1 -- 172.21.15.96:0/4039146499 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5610052150 con 0x7f56200a98f0 2024-03-25T17:29:33.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.329+0000 7f561cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:33.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.329+0000 7f561cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:33.331 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:33.331 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.331+0000 7f56267b5640 10 librados: watch_flush enter 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.331+0000 7f56267b5640 10 librados: watch_flush exit 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.331+0000 7f56267b5640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.331+0000 7f56267b5640 10 client.4221.objecter successfully canceled tick 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 >> v1:172.21.15.203:6800/3611964061 conn(0x7f560c06dfb0 legacy=0x7f560c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 10 monclient: shutdown 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 >> v1:172.21.15.96:6789/0 conn(0x7f56200a98f0 legacy=0x7f56201376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:33.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 shutdown_connections 2024-03-25T17:29:33.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 >> 172.21.15.96:0/4039146499 conn(0x7f56200b7ea0 msgr2=0x7f562013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:33.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 shutdown_connections 2024-03-25T17:29:33.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 -- 172.21.15.96:0/4039146499 wait complete. 2024-03-25T17:29:33.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:33.332+0000 7f56267b5640 1 librados: shutdown 2024-03-25T17:29:33.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:34.656 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:26.377 35578== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:34.701 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:26.422 35579== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:34.848 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:26.593 35576== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:34.947 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:00:26.675 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:34.977 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:00:26.725 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.229 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:00:26.977 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.253 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:00:26.990 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.720 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:27.459 35577== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:40.151 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:40.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.263+0000 7f36e9f88640 10 monclient: get_monmap_and_config 2024-03-25T17:29:40.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.263+0000 7f36e9f88640 10 monclient: build_initial_monmap 2024-03-25T17:29:40.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.263+0000 7f36e9f88640 10 monclient: monmap: 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:40.264987+0000 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:40.264987+0000 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.264+0000 7f36e9f88640 1 Processor -- start 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.264+0000 7f36e9f88640 1 -- start start 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.264+0000 7f36e9f88640 10 monclient: init 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-c con 0x7f36e40b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-a con 0x7f36e40b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-b con 0x7f36e414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40af940 con 0x7f36e40b1730 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40b0a70 con 0x7f36e40b4d60 2024-03-25T17:29:40.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40b0d50 con 0x7f36e414be90 2024-03-25T17:29:40.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:40.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e9f88640 10 monclient(hunting): authenticate will time out at 5338.547256s 2024-03-25T17:29:40.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e3fff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f36e414be90 0x7f36e414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:54836/0 (socket says 172.21.15.96:54836) 2024-03-25T17:29:40.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.265+0000 7f36e3fff640 1 -- 172.21.15.96:0/3437281607 learned_addr learned my addr 172.21.15.96:0/3437281607 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:40.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774294383 0 0) 0x7f36e40b0d50 con 0x7f36e414be90 2024-03-25T17:29:40.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): my global_id is 4223 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc003520 con 0x7f36e414be90 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384620289 0 0) 0x7f36e40b0a70 con 0x7f36e40b4d60 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T17:29:40.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.268+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e40b0d50 con 0x7f36e40b4d60 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750114696 0 0) 0x7f36e40af940 con 0x7f36e40b1730 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T17:29:40.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.269+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e40b0a70 con 0x7f36e40b1730 2024-03-25T17:29:40.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866849932 0 0) 0x7f36e40b0d50 con 0x7f36e40b4d60 2024-03-25T17:29:40.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 >> v1:172.21.15.96:6790/0 conn(0x7f36e414be90 legacy=0x7f36e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 >> v1:172.21.15.96:6789/0 conn(0x7f36e40b1730 legacy=0x7f36e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 monclient: found mon.noname-c 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e40b1030 con 0x7f36e40b4d60 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f36d4001630 con 0x7f36e40b4d60 2024-03-25T17:29:40.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e27fc640 10 monclient: dump: 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.282 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e9f88640 5 monclient: authenticate success, global_id 4216 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.272+0000 7f36e9f88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36d4001880 con 0x7f36e40b4d60 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e9f88640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e9f88640 4 set_mon_vals no callback set 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 1 -- 172.21.15.96:0/3437281607 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f36d4001c10 con 0x7f36e40b4d60 2024-03-25T17:29:40.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.274+0000 7f36e27fc640 10 monclient: dump: 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:40.284 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 10 monclient: shutdown 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 1 -- 172.21.15.96:0/3437281607 >> v1:172.21.15.203:6789/0 conn(0x7f36e40b4d60 legacy=0x7f36e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 1 -- 172.21.15.96:0/3437281607 shutdown_connections 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 1 -- 172.21.15.96:0/3437281607 >> 172.21.15.96:0/3437281607 conn(0x7f36e40617a0 msgr2=0x7f36e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 1 -- 172.21.15.96:0/3437281607 shutdown_connections 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.275+0000 7f36e9f88640 1 -- 172.21.15.96:0/3437281607 wait complete. 2024-03-25T17:29:40.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.276+0000 7f36e9f88640 10 monclient: build_initial_monmap 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.276+0000 7f36e9f88640 10 monclient: monmap: 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:40.286 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.276+0000 7f36e9f88640 1 librados: starting msgr at 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.276+0000 7f36e9f88640 1 librados: starting objecter 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.276+0000 7f36e9f88640 1 Processor -- start 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.277+0000 7f36e9f88640 1 -- start start 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.277+0000 7f36e9f88640 1 librados: setting wanted keys 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.277+0000 7f36e9f88640 1 librados: calling monclient init 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.277+0000 7f36e9f88640 10 monclient: init 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:40.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-b con 0x7f36e40b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-a con 0x7f36e40b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient(hunting): picked mon.noname-c con 0x7f36e414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e41daa80 con 0x7f36e40b4d60 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e413b800 con 0x7f36e40b1730 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e4139130 con 0x7f36e414be90 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e2ffd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f36e40b4d60 0x7f36e4132e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42390/0 (socket says 172.21.15.96:42390) 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e2ffd640 1 -- 172.21.15.96:0/3481255541 learned_addr learned my addr 172.21.15.96:0/3481255541 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.278+0000 7f36e9f88640 10 monclient(hunting): authenticate will time out at 5338.560914s 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter ms_handle_connect 0x7f36e40b4d60 2024-03-25T17:29:40.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:40.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter ms_handle_connect 0x7f36e40b1730 2024-03-25T17:29:40.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:40.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter ms_handle_connect 0x7f36e414be90 2024-03-25T17:29:40.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.279+0000 7f36cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242596266 0 0) 0x7f36e413b800 con 0x7f36e40b1730 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 10 monclient(hunting): my global_id is 4219 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.281+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d0004eb0 con 0x7f36e40b1730 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227132124 0 0) 0x7f36e4139130 con 0x7f36e414be90 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): my global_id is 4226 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e413b800 con 0x7f36e414be90 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960340934 0 0) 0x7f36e41daa80 con 0x7f36e40b4d60 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 10 monclient(hunting): my global_id is 4227 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.282+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4139130 con 0x7f36e40b4d60 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.285+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217697294 0 0) 0x7f36e413b800 con 0x7f36e414be90 2024-03-25T17:29:40.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e41daa80 con 0x7f36e414be90 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f36dc0019e0 con 0x7f36e414be90 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 10 monclient(hunting): dump: 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.297 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.286+0000 7f36cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611586407 0 0) 0x7f36d0004eb0 con 0x7f36e40b1730 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e413b800 con 0x7f36e40b1730 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f36d4001b90 con 0x7f36e40b1730 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.288+0000 7f36cbfff640 10 monclient(hunting): dump: 2024-03-25T17:29:40.298 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093318539 0 0) 0x7f36e41daa80 con 0x7f36e414be90 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 >> v1:172.21.15.203:6789/0 conn(0x7f36e40b1730 legacy=0x7f36e4139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.299 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 >> v1:172.21.15.96:6789/0 conn(0x7f36e40b4d60 legacy=0x7f36e4132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 1 monclient: found mon.c 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e413d0f0 con 0x7f36e414be90 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 10 monclient: _finish_auth 0 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36e9f88640 5 monclient: authenticate success, global_id 4226 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36e9f88640 10 monclient: _renew_subs 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36e9f88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:40.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.290+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e413a2b0 con 0x7f36e414be90 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.291+0000 7f36e9f88640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.291+0000 7f36e9f88640 10 monclient: _renew_subs 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.291+0000 7f36e9f88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.291+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e4133710 con 0x7f36e414be90 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.291+0000 7f36e9f88640 1 librados: init done 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.292+0000 7f36e9f88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.292+0000 7f36e9f88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.292+0000 7f36e9f88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.292+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e413b640 con 0x7f36e414be90 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.293+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36dc001ef0 con 0x7f36e414be90 2024-03-25T17:29:40.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.293+0000 7f36cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:40.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.293+0000 7f36e1ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.317+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f36dc004d00 con 0x7f36e414be90 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.317+0000 7f36cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.317+0000 7f36cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.317+0000 7f36cbfff640 10 monclient: dump: 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:40.318 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:40.319 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:40.319 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:40.319 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:40.319 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:40.319 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:40.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.317+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f36dc01acd0 con 0x7f36e414be90 2024-03-25T17:29:40.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.318+0000 7f36e37fe640 10 monclient: get_auth_request con 0x7f36d006df70 auth_method 0 2024-03-25T17:29:40.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.318+0000 7f36cbfff640 10 client.4226.objecter ms_handle_connect 0x7f36d006df70 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f36dc054800 con 0x7f36e414be90 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 10 client.4226.objecter ms_dispatch 0x7f36e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T17:29:40.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36dc050310 con 0x7f36e414be90 2024-03-25T17:29:40.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:40.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.603+0000 7f36cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:40.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.791+0000 7f36e9f88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:40.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.791+0000 7f36e9f88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:40.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.791+0000 7f36e9f88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:40.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.791+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36e40b5e80 con 0x7f36e414be90 2024-03-25T17:29:40.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.927+0000 7f36cbfff640 1 -- 172.21.15.96:0/3481255541 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f36dc052240 con 0x7f36e414be90 2024-03-25T17:29:40.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.927+0000 7f36cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:40.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.927+0000 7f36cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:40.928 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:40.928 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.929+0000 7f36e9f88640 10 librados: watch_flush enter 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.929+0000 7f36e9f88640 10 librados: watch_flush exit 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.929+0000 7f36e9f88640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.929+0000 7f36e9f88640 10 client.4226.objecter successfully canceled tick 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 >> v1:172.21.15.203:6800/3611964061 conn(0x7f36d006df70 legacy=0x7f36d0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 10 monclient: shutdown 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:40.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 >> v1:172.21.15.96:6790/0 conn(0x7f36e414be90 legacy=0x7f36e41daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:40.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 shutdown_connections 2024-03-25T17:29:40.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 >> 172.21.15.96:0/3481255541 conn(0x7f36e40617a0 msgr2=0x7f36e413c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:40.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.930+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 shutdown_connections 2024-03-25T17:29:40.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.931+0000 7f36e9f88640 1 -- 172.21.15.96:0/3481255541 wait complete. 2024-03-25T17:29:40.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:40.931+0000 7f36e9f88640 1 librados: shutdown 2024-03-25T17:29:40.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:43.296 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-25T17:29:43.293+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T17:29:43.440 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-25T17:29:43.437+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T17:29:43.593 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-03-25T17:29:43.590+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T17:29:43.713 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-03-25T17:29:43.709+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T17:29:43.763 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-25T17:29:43.760+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T17:29:43.771 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-03-25T17:29:43.768+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T17:29:43.845 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-03-25T17:29:43.842+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T17:29:44.213 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-25T17:29:44.210+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T17:29:47.752 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:47.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.869+0000 7fea89afb640 10 monclient: get_monmap_and_config 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.869+0000 7fea89afb640 10 monclient: build_initial_monmap 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.869+0000 7fea89afb640 10 monclient: monmap: 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:47.871211+0000 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:47.871211+0000 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:47.871 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:47.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.870+0000 7fea89afb640 1 Processor -- start 2024-03-25T17:29:47.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 1 -- start start 2024-03-25T17:29:47.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 10 monclient: init 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-a con 0x7fea840a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.871+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-c con 0x7fea8413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-b con 0x7fea841430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea84142520 con 0x7fea840a98f0 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840b28a0 con 0x7fea8413f710 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840b2a30 con 0x7fea841430c0 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:47.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.872+0000 7fea89afb640 10 monclient(hunting): authenticate will time out at 5346.154030s 2024-03-25T17:29:47.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.882+0000 7fea82ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fea8413f710 0x7fea841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:39276/0 (socket says 172.21.15.96:39276) 2024-03-25T17:29:47.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.882+0000 7fea82ffd640 1 -- 172.21.15.96:0/645588955 learned_addr learned my addr 172.21.15.96:0/645588955 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797030807 0 0) 0x7fea840b28a0 con 0x7fea8413f710 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 10 monclient(hunting): my global_id is 4246 2024-03-25T17:29:47.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.899+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003520 con 0x7fea8413f710 2024-03-25T17:29:47.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822819293 0 0) 0x7fea840b2a30 con 0x7fea841430c0 2024-03-25T17:29:47.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:47.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:47.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:47.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 10 monclient(hunting): my global_id is 4253 2024-03-25T17:29:47.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.906+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea840b28a0 con 0x7fea841430c0 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238543078 0 0) 0x7fea84142520 con 0x7fea840a98f0 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 10 monclient(hunting): my global_id is 4254 2024-03-25T17:29:47.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.911+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea840b2a30 con 0x7fea840a98f0 2024-03-25T17:29:47.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 302506264 0 0) 0x7fea840b28a0 con 0x7fea841430c0 2024-03-25T17:29:47.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 >> v1:172.21.15.203:6789/0 conn(0x7fea8413f710 legacy=0x7fea841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:47.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 >> v1:172.21.15.96:6789/0 conn(0x7fea840a98f0 legacy=0x7fea840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:47.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 monclient: found mon.noname-b 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840b44c0 con 0x7fea841430c0 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fea7c0019f0 con 0x7fea841430c0 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea827fc640 10 monclient: dump: 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.916 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea89afb640 5 monclient: authenticate success, global_id 4253 2024-03-25T17:29:47.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.914+0000 7fea89afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:47.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea7c001b50 con 0x7fea841430c0 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 1 -- 172.21.15.96:0/645588955 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fea7c001f60 con 0x7fea841430c0 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea89afb640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea89afb640 4 set_mon_vals no callback set 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.931+0000 7fea827fc640 10 monclient: dump: 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:47.932 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:47.933 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:47.933 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:47.933 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 10 monclient: shutdown 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 1 -- 172.21.15.96:0/645588955 >> v1:172.21.15.96:6790/0 conn(0x7fea841430c0 legacy=0x7fea8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 1 -- 172.21.15.96:0/645588955 shutdown_connections 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 1 -- 172.21.15.96:0/645588955 >> 172.21.15.96:0/645588955 conn(0x7fea840b7ea0 msgr2=0x7fea840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 1 -- 172.21.15.96:0/645588955 shutdown_connections 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 1 -- 172.21.15.96:0/645588955 wait complete. 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 10 monclient: build_initial_monmap 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.932+0000 7fea89afb640 10 monclient: monmap: 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:47.935 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 librados: starting msgr at 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 librados: starting objecter 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 Processor -- start 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 -- start start 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 librados: setting wanted keys 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 1 librados: calling monclient init 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.933+0000 7fea89afb640 10 monclient: init 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:47.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-a con 0x7fea840a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-c con 0x7fea8413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient(hunting): picked mon.noname-b con 0x7fea841430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841df010 con 0x7fea840a98f0 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841dfc40 con 0x7fea841430c0 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841dfdd0 con 0x7fea8413f710 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:47.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.934+0000 7fea89afb640 10 monclient(hunting): authenticate will time out at 5346.216343s 2024-03-25T17:29:47.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.942+0000 7fea82ffd640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fea8413f710 0x7fea8406ead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:53214/0 (socket says 172.21.15.96:53214) 2024-03-25T17:29:47.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.942+0000 7fea82ffd640 1 -- 172.21.15.96:0/1852324491 learned_addr learned my addr 172.21.15.96:0/1852324491 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:47.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.943+0000 7fea6bfff640 10 client.?.objecter ms_handle_connect 0x7fea8413f710 2024-03-25T17:29:47.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.943+0000 7fea6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:47.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.948+0000 7fea6bfff640 10 client.?.objecter ms_handle_connect 0x7fea841430c0 2024-03-25T17:29:47.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.948+0000 7fea6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286843869 0 0) 0x7fea841dfdd0 con 0x7fea8413f710 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 10 monclient(hunting): my global_id is 4256 2024-03-25T17:29:47.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.952+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea70004eb0 con 0x7fea8413f710 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058732140 0 0) 0x7fea841dfc40 con 0x7fea841430c0 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 monclient(hunting): my global_id is 4249 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea841dfdd0 con 0x7fea841430c0 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 client.?.objecter ms_handle_connect 0x7fea840a98f0 2024-03-25T17:29:47.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.957+0000 7fea6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1972020375 0 0) 0x7fea70004eb0 con 0x7fea8413f710 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea841dfc40 con 0x7fea8413f710 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fea78001950 con 0x7fea8413f710 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 10 monclient(hunting): dump: 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.965 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:47.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.964+0000 7fea6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3923963846 0 0) 0x7fea841dfc40 con 0x7fea8413f710 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 >> v1:172.21.15.203:6789/0 conn(0x7fea841430c0 legacy=0x7fea84138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 >> v1:172.21.15.96:6789/0 conn(0x7fea840a98f0 legacy=0x7fea841df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 1 monclient: found mon.c 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:47.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8406dee0 con 0x7fea8413f710 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 10 monclient: _finish_auth 0 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 5 monclient: authenticate success, global_id 4256 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 10 monclient: _renew_subs 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8406e070 con 0x7fea8413f710 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 10 monclient: _renew_subs 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84139f00 con 0x7fea8413f710 2024-03-25T17:29:47.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.974+0000 7fea89afb640 1 librados: init done 2024-03-25T17:29:47.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.975+0000 7fea89afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:47.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.975+0000 7fea89afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:47.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.975+0000 7fea89afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:47.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:47.975+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea840627b0 con 0x7fea8413f710 2024-03-25T17:29:48.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea780047e0 con 0x7fea8413f710 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fea78001d80 con 0x7fea8413f710 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea6bfff640 10 monclient: dump: 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:48.265 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:48.266 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:48.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.263+0000 7fea81ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:48.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.264+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fea78004c30 con 0x7fea8413f710 2024-03-25T17:29:48.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.265+0000 7fea837fe640 10 monclient: get_auth_request con 0x7fea7006dd10 auth_method 0 2024-03-25T17:29:48.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.265+0000 7fea6bfff640 10 client.4256.objecter ms_handle_connect 0x7fea7006dd10 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fea78054810 con 0x7fea8413f710 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 10 client.4256.objecter ms_dispatch 0x7fea84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea78050150 con 0x7fea8413f710 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:48.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.270+0000 7fea6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:48.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.527+0000 7fea89afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:48.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.527+0000 7fea89afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:48.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.527+0000 7fea89afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:29:48.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.527+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea841df1a0 con 0x7fea8413f710 2024-03-25T17:29:48.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.554+0000 7fea6bfff640 1 -- 172.21.15.96:0/1852324491 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fea78052250 con 0x7fea8413f710 2024-03-25T17:29:48.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.554+0000 7fea6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:48.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.554+0000 7fea6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:48.555 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:48.555 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 10 librados: watch_flush enter 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 10 librados: watch_flush exit 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 10 client.4256.objecter successfully canceled tick 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 >> v1:172.21.15.203:6800/3611964061 conn(0x7fea7006dd10 legacy=0x7fea700701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 10 monclient: shutdown 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:48.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 >> v1:172.21.15.96:6790/0 conn(0x7fea8413f710 legacy=0x7fea8406ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:48.559 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 shutdown_connections 2024-03-25T17:29:48.559 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 >> 172.21.15.96:0/1852324491 conn(0x7fea840b7ea0 msgr2=0x7fea84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:48.559 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.556+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 shutdown_connections 2024-03-25T17:29:48.559 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.557+0000 7fea89afb640 1 -- 172.21.15.96:0/1852324491 wait complete. 2024-03-25T17:29:48.559 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:48.557+0000 7fea89afb640 1 librados: shutdown 2024-03-25T17:29:48.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:55.374 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:55.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.490+0000 7f5237aba640 10 monclient: get_monmap_and_config 2024-03-25T17:29:55.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.490+0000 7f5237aba640 10 monclient: build_initial_monmap 2024-03-25T17:29:55.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.490+0000 7f5237aba640 10 monclient: monmap: 2024-03-25T17:29:55.491 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:55.491 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:29:55.492163+0000 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:29:55.492163+0000 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:29:55.492 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.491+0000 7f5237aba640 1 Processor -- start 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.491+0000 7f5237aba640 1 -- start start 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.491+0000 7f5237aba640 10 monclient: init 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-a con 0x7f52300a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-b con 0x7f523013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-c con 0x7f52301430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5230142520 con 0x7f52300a98f0 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300b28a0 con 0x7f52301430c0 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300b2a30 con 0x7f523013f710 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:55.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.492+0000 7f5237aba640 10 monclient(hunting): authenticate will time out at 5353.774355s 2024-03-25T17:29:55.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.499+0000 7f5236057640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f52301430c0 0x7f523014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:44838/0 (socket says 172.21.15.96:44838) 2024-03-25T17:29:55.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.499+0000 7f5236057640 1 -- 172.21.15.96:0/239991256 learned_addr learned my addr 172.21.15.96:0/239991256 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:55.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362275357 0 0) 0x7f52300b28a0 con 0x7f52301430c0 2024-03-25T17:29:55.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 10 monclient(hunting): _negotiate 2024-03-25T17:29:55.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:55.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:55.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 10 monclient(hunting): my global_id is 4252 2024-03-25T17:29:55.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.507+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c003520 con 0x7f52301430c0 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534095249 0 0) 0x7f521c003520 con 0x7f52301430c0 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 >> v1:172.21.15.96:6790/0 conn(0x7f523013f710 legacy=0x7f52301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 >> v1:172.21.15.96:6789/0 conn(0x7f52300a98f0 legacy=0x7f52300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: _finish_hunting 0 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 monclient: found mon.noname-c 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300b44c0 con 0x7f52301430c0 2024-03-25T17:29:55.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: _finish_auth 0 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: _check_auth_tickets 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5237aba640 5 monclient: authenticate success, global_id 4252 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5228001a10 con 0x7f52301430c0 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5234854640 10 monclient: dump: 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.515 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:55.516 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:55.516 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:55.516 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.513+0000 7f5237aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5228001be0 con 0x7f52301430c0 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 1 -- 172.21.15.96:0/239991256 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5228002030 con 0x7f52301430c0 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5237aba640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5237aba640 4 set_mon_vals no callback set 2024-03-25T17:29:55.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.524+0000 7f5234854640 10 monclient: dump: 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:55.526 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 10 monclient: shutdown 2024-03-25T17:29:55.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:55.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 1 -- 172.21.15.96:0/239991256 >> v1:172.21.15.203:6789/0 conn(0x7f52301430c0 legacy=0x7f523014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:55.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 1 -- 172.21.15.96:0/239991256 shutdown_connections 2024-03-25T17:29:55.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 1 -- 172.21.15.96:0/239991256 >> 172.21.15.96:0/239991256 conn(0x7f52300b7ea0 msgr2=0x7f52300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 1 -- 172.21.15.96:0/239991256 shutdown_connections 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.525+0000 7f5237aba640 1 -- 172.21.15.96:0/239991256 wait complete. 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 10 monclient: build_initial_monmap 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 10 monclient: monmap: 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:29:55.529 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 librados: starting msgr at 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 librados: starting objecter 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 Processor -- start 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 -- start start 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 librados: setting wanted keys 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 1 librados: calling monclient init 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.526+0000 7f5237aba640 10 monclient: init 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-a con 0x7f52300a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-b con 0x7f523013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient(hunting): picked mon.noname-c con 0x7f52301430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5230138100 con 0x7f52300a98f0 2024-03-25T17:29:55.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300abcc0 con 0x7f523013f710 2024-03-25T17:29:55.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52301d8320 con 0x7f52301430c0 2024-03-25T17:29:55.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:55.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.527+0000 7f5237aba640 10 monclient(hunting): authenticate will time out at 5353.809397s 2024-03-25T17:29:55.533 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.533+0000 7f5235856640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f52300a98f0 0x7f52301386b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34092/0 (socket says 172.21.15.96:34092) 2024-03-25T17:29:55.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.533+0000 7f5235856640 1 -- 172.21.15.96:0/1579251307 learned_addr learned my addr 172.21.15.96:0/1579251307 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:29:55.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.534+0000 7f52267fc640 10 client.?.objecter ms_handle_connect 0x7f52300a98f0 2024-03-25T17:29:55.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.534+0000 7f52267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175645444 0 0) 0x7f5230138100 con 0x7f52300a98f0 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5214004eb0 con 0x7f52300a98f0 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 client.?.objecter ms_handle_connect 0x7f52301430c0 2024-03-25T17:29:55.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.542+0000 7f52267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:55.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.549+0000 7f52267fc640 10 client.?.objecter ms_handle_connect 0x7f523013f710 2024-03-25T17:29:55.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.549+0000 7f52267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:55.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.553+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058256963 0 0) 0x7f52300abcc0 con 0x7f523013f710 2024-03-25T17:29:55.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.554+0000 7f52267fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.554+0000 7f52267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.554+0000 7f52267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.554+0000 7f52267fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.554+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5230138100 con 0x7f523013f710 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598670791 0 0) 0x7f52301d8320 con 0x7f52301430c0 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:55.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:55.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): my global_id is 4259 2024-03-25T17:29:55.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52300abcc0 con 0x7f52301430c0 2024-03-25T17:29:55.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441670328 0 0) 0x7f5214004eb0 con 0x7f52300a98f0 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52301d8320 con 0x7f52300a98f0 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f522c001960 con 0x7f52300a98f0 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): dump: 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:55.557 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:55.558 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.555+0000 7f52267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 178440328 0 0) 0x7f5230138100 con 0x7f523013f710 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5214004eb0 con 0x7f523013f710 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f52200018b0 con 0x7f523013f710 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient(hunting): dump: 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.568 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:55.569 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:55.569 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:55.569 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2975414182 0 0) 0x7f52301d8320 con 0x7f52300a98f0 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 >> v1:172.21.15.96:6790/0 conn(0x7f52301430c0 legacy=0x7f52301e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 >> v1:172.21.15.203:6789/0 conn(0x7f523013f710 legacy=0x7f52301d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 monclient: found mon.a 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52301350d0 con 0x7f52300a98f0 2024-03-25T17:29:55.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f52267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.567+0000 7f5237aba640 5 monclient: authenticate success, global_id 4260 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 10 monclient: _renew_subs 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52301353a0 con 0x7f52300a98f0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 10 monclient: _renew_subs 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5230137b60 con 0x7f52300a98f0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.568+0000 7f5237aba640 1 librados: init done 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.569+0000 7f5237aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.569+0000 7f5237aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.569+0000 7f5237aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:55.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.569+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230137eb0 con 0x7f52300a98f0 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f522c0035f0 con 0x7f52300a98f0 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f522c003940 con 0x7f52300a98f0 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f52267fc640 10 monclient: dump: 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:29:55.868 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:29:55.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.867+0000 7f5227fff640 4 set_mon_vals no callback set 2024-03-25T17:29:55.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.868+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f522c003b60 con 0x7f52300a98f0 2024-03-25T17:29:55.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.869+0000 7f5235055640 10 monclient: get_auth_request con 0x7f521406df70 auth_method 0 2024-03-25T17:29:55.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.869+0000 7f52267fc640 10 client.4260.objecter ms_handle_connect 0x7f521406df70 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f522c054750 con 0x7f52300a98f0 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 10 client.4260.objecter ms_dispatch 0x7f5230000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f522c050260 con 0x7f52300a98f0 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:55.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:55.874+0000 7f52267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:56.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.100+0000 7f5237aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:56.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.100+0000 7f5237aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:56.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.100+0000 7f5237aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:29:56.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.100+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52300627b0 con 0x7f52300a98f0 2024-03-25T17:29:56.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.131+0000 7f52267fc640 1 -- 172.21.15.96:0/1579251307 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f522c052190 con 0x7f52300a98f0 2024-03-25T17:29:56.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.131+0000 7f52267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:56.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.131+0000 7f52267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:56.132 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:29:56.132 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 10 librados: watch_flush enter 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 10 librados: watch_flush exit 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 10 client.4260.objecter successfully canceled tick 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 >> v1:172.21.15.203:6800/3611964061 conn(0x7f521406df70 legacy=0x7f5214070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 10 monclient: shutdown 2024-03-25T17:29:56.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 >> v1:172.21.15.96:6789/0 conn(0x7f52300a98f0 legacy=0x7f52301386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 shutdown_connections 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 >> 172.21.15.96:0/1579251307 conn(0x7f52300b7ea0 msgr2=0x7f5230074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 shutdown_connections 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 -- 172.21.15.96:0/1579251307 wait complete. 2024-03-25T17:29:56.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:29:56.134+0000 7f5237aba640 1 librados: shutdown 2024-03-25T17:29:56.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:59.877 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:24.103 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:59.938 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:24.164 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.080 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:24.295 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.082 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:24.308 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.140 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:24.366 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.140 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:24.356 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.164 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:24.391 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.356 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:24.571 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:00.374 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:24.600 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:02.952 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:03.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.066+0000 7fb502756640 10 monclient: get_monmap_and_config 2024-03-25T17:30:03.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.066+0000 7fb502756640 10 monclient: build_initial_monmap 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.066+0000 7fb502756640 10 monclient: monmap: 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:03.066382+0000 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:03.066382+0000 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:03.066 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.067 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 1 Processor -- start 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 1 -- start start 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient: init 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient(hunting): picked mon.noname-a con 0x7fb4fc0705b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient(hunting): picked mon.noname-b con 0x7fb4fc1402d0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient(hunting): picked mon.noname-c con 0x7fb4fc149ae0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0718c0 con 0x7fb4fc0705b0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0714c0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc072b50 con 0x7fb4fc1402d0 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:03.068 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.067+0000 7fb502756640 10 monclient(hunting): authenticate will time out at 5361.348738s 2024-03-25T17:30:03.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.086+0000 7fb4fbfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb4fc0705b0 0x7fb4fc13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38110/0 (socket says 172.21.15.96:38110) 2024-03-25T17:30:03.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.086+0000 7fb4fbfff640 1 -- 172.21.15.96:0/4002186330 learned_addr learned my addr 172.21.15.96:0/4002186330 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:03.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689920276 0 0) 0x7fb4fc0714c0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T17:30:03.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.099+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4003520 con 0x7fb4fc149ae0 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396095003 0 0) 0x7fb4fc0718c0 con 0x7fb4fc0705b0 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 10 monclient(hunting): my global_id is 4263 2024-03-25T17:30:03.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.105+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc0714c0 con 0x7fb4fc0705b0 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811172887 0 0) 0x7fb4fc072b50 con 0x7fb4fc1402d0 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T17:30:03.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.108+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc0718c0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4137169848 0 0) 0x7fb4fc0718c0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 >> v1:172.21.15.203:6789/0 conn(0x7fb4fc149ae0 legacy=0x7fb4fc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 >> v1:172.21.15.96:6789/0 conn(0x7fb4fc0705b0 legacy=0x7fb4fc13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: _finish_hunting 0 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 monclient: found mon.noname-b 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc13dd10 con 0x7fb4fc1402d0 2024-03-25T17:30:03.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: _finish_auth 0 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: _check_auth_tickets 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb4f00015b0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb4faffd640 10 monclient: dump: 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:03.120 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb502756640 5 monclient: authenticate success, global_id 4262 2024-03-25T17:30:03.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.119+0000 7fb502756640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:03.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4f0001710 con 0x7fb4fc1402d0 2024-03-25T17:30:03.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 1 -- 172.21.15.96:0/4002186330 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb4f0001b20 con 0x7fb4fc1402d0 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb502756640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb502756640 4 set_mon_vals no callback set 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.148+0000 7fb4faffd640 10 monclient: dump: 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:03.149 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 10 monclient: shutdown 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 1 -- 172.21.15.96:0/4002186330 >> v1:172.21.15.96:6790/0 conn(0x7fb4fc1402d0 legacy=0x7fb4fc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 1 -- 172.21.15.96:0/4002186330 shutdown_connections 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 1 -- 172.21.15.96:0/4002186330 >> 172.21.15.96:0/4002186330 conn(0x7fb4fc0b5d10 msgr2=0x7fb4fc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:03.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 1 -- 172.21.15.96:0/4002186330 shutdown_connections 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.149+0000 7fb502756640 1 -- 172.21.15.96:0/4002186330 wait complete. 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.150+0000 7fb502756640 10 monclient: build_initial_monmap 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.150+0000 7fb502756640 10 monclient: monmap: 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:03.152 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.150+0000 7fb502756640 1 librados: starting msgr at 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.150+0000 7fb502756640 1 librados: starting objecter 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.150+0000 7fb502756640 1 Processor -- start 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 1 -- start start 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 1 librados: setting wanted keys 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 1 librados: calling monclient init 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 10 monclient: init 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 10 monclient(hunting): picked mon.noname-a con 0x7fb4fc0705b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.151+0000 7fb502756640 10 monclient(hunting): picked mon.noname-b con 0x7fb4fc1402d0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 10 monclient(hunting): picked mon.noname-c con 0x7fb4fc149ae0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc1d5410 con 0x7fb4fc0705b0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc13f690 con 0x7fb4fc1402d0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc13e4a0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:03.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.152+0000 7fb502756640 10 monclient(hunting): authenticate will time out at 5361.433004s 2024-03-25T17:30:03.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.159+0000 7fb4fb7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb4fc1402d0 0x7fb4fc1d5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:49168/0 (socket says 172.21.15.96:49168) 2024-03-25T17:30:03.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.159+0000 7fb4fb7fe640 1 -- 172.21.15.96:0/2737016059 learned_addr learned my addr 172.21.15.96:0/2737016059 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:03.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.164+0000 7fb4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb4fc1402d0 2024-03-25T17:30:03.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.164+0000 7fb4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:03.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.165+0000 7fb4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb4fc149ae0 2024-03-25T17:30:03.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.165+0000 7fb4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:03.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.167+0000 7fb4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb4fc0705b0 2024-03-25T17:30:03.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.167+0000 7fb4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:03.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428693661 0 0) 0x7fb4fc13e4a0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): my global_id is 4265 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e8004eb0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829811245 0 0) 0x7fb4fc13f690 con 0x7fb4fc1402d0 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 10 monclient(hunting): my global_id is 4261 2024-03-25T17:30:03.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.175+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc13e4a0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694318505 0 0) 0x7fb4fc1d5410 con 0x7fb4fc0705b0 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 10 monclient(hunting): my global_id is 4266 2024-03-25T17:30:03.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.187+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc13f690 con 0x7fb4fc0705b0 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820739778 0 0) 0x7fb4fc13e4a0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4fc1d5410 con 0x7fb4fc1402d0 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb4f0001a60 con 0x7fb4fc1402d0 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 10 monclient(hunting): dump: 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.189 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.189+0000 7fb4f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:03.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2801741866 0 0) 0x7fb4e8004eb0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4fc13e4a0 con 0x7fb4fc149ae0 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb4f4001b30 con 0x7fb4fc149ae0 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.192+0000 7fb4f8ff9640 10 monclient(hunting): dump: 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:03.193 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:03.194 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:03.194 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 884484046 0 0) 0x7fb4fc1d5410 con 0x7fb4fc1402d0 2024-03-25T17:30:03.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 >> v1:172.21.15.96:6790/0 conn(0x7fb4fc149ae0 legacy=0x7fb4fc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 >> v1:172.21.15.96:6789/0 conn(0x7fb4fc0705b0 legacy=0x7fb4fc13ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 1 monclient: found mon.b 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc1d6040 con 0x7fb4fc1402d0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 10 monclient: _finish_auth 0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb4f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 5 monclient: authenticate success, global_id 4261 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 10 monclient: _renew_subs 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc13fea0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 10 monclient: _renew_subs 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc132770 con 0x7fb4fc1402d0 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.202+0000 7fb502756640 1 librados: init done 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.203+0000 7fb502756640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.203+0000 7fb502756640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:03.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.203+0000 7fb502756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:03.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.203+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc1d55a0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4f00038a0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb4f0003cb0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4f8ff9640 10 monclient: dump: 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:03.501 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:03.502 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.501+0000 7fb4fa7fc640 4 set_mon_vals no callback set 2024-03-25T17:30:03.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.502+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fb4f0003ed0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.503+0000 7fb4fbfff640 10 monclient: get_auth_request con 0x7fb4e806df70 auth_method 0 2024-03-25T17:30:03.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.503+0000 7fb4f8ff9640 10 client.4261.objecter ms_handle_connect 0x7fb4e806df70 2024-03-25T17:30:03.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.508+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fb4f00548b0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.508+0000 7fb4f8ff9640 10 client.4261.objecter ms_dispatch 0x7fb4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:03.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.508+0000 7fb4f8ff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:03.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.508+0000 7fb4f8ff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:03.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.508+0000 7fb4f8ff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T17:30:03.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.510+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4f00503c0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.510+0000 7fb4f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:03.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.510+0000 7fb4f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:03.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.763+0000 7fb502756640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:03.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.763+0000 7fb502756640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:03.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.763+0000 7fb502756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:03.763 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.763+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4fc0627b0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.823+0000 7fb4f8ff9640 1 -- 172.21.15.96:0/2737016059 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fb4f00522f0 con 0x7fb4fc1402d0 2024-03-25T17:30:03.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.823+0000 7fb4f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:03.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.823+0000 7fb4f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:03.823 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:03.824 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 10 librados: watch_flush enter 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 10 librados: watch_flush exit 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 10 client.4261.objecter successfully canceled tick 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 >> v1:172.21.15.203:6800/3611964061 conn(0x7fb4e806df70 legacy=0x7fb4e8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 10 monclient: shutdown 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 >> v1:172.21.15.203:6789/0 conn(0x7fb4fc1402d0 legacy=0x7fb4fc1d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 shutdown_connections 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 >> 172.21.15.96:0/2737016059 conn(0x7fb4fc0b5d10 msgr2=0x7fb4fc0b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 shutdown_connections 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 -- 172.21.15.96:0/2737016059 wait complete. 2024-03-25T17:30:03.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:03.826+0000 7fb502756640 1 librados: shutdown 2024-03-25T17:30:03.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:10.644 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:10.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.760+0000 7fc0376e4640 10 monclient: get_monmap_and_config 2024-03-25T17:30:10.762 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.760+0000 7fc0376e4640 10 monclient: build_initial_monmap 2024-03-25T17:30:10.762 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.760+0000 7fc0376e4640 10 monclient: monmap: 2024-03-25T17:30:10.762 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:10.762 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:10.762 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:10.762203+0000 2024-03-25T17:30:10.763 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:10.762203+0000 2024-03-25T17:30:10.763 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:10.763 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:10.763 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:10.763 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:10.764 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:10.764 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:10.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.761+0000 7fc0376e4640 1 Processor -- start 2024-03-25T17:30:10.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.761+0000 7fc0376e4640 1 -- start start 2024-03-25T17:30:10.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.761+0000 7fc0376e4640 10 monclient: init 2024-03-25T17:30:10.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:10.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:30:10.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc0300a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc03013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc0301430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030142520 con 0x7fc0300a98f0 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300b28a0 con 0x7fc0301430c0 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300b2a30 con 0x7fc03013f710 2024-03-25T17:30:10.768 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:10.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.762+0000 7fc0376e4640 10 monclient(hunting): authenticate will time out at 5369.044511s 2024-03-25T17:30:10.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.770+0000 7fc034c7f640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fc03013f710 0x7fc0301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:34254/0 (socket says 172.21.15.96:34254) 2024-03-25T17:30:10.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.770+0000 7fc034c7f640 1 -- 172.21.15.96:0/722738141 learned_addr learned my addr 172.21.15.96:0/722738141 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:10.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892383726 0 0) 0x7fc0300b2a30 con 0x7fc03013f710 2024-03-25T17:30:10.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:10.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:10.781 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:10.781 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T17:30:10.781 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.779+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01c003520 con 0x7fc03013f710 2024-03-25T17:30:10.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040477721 0 0) 0x7fc0300b28a0 con 0x7fc0301430c0 2024-03-25T17:30:10.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:10.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:10.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:10.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 10 monclient(hunting): my global_id is 4264 2024-03-25T17:30:10.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.786+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0300b2a30 con 0x7fc0301430c0 2024-03-25T17:30:10.810 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730640531 0 0) 0x7fc030142520 con 0x7fc0300a98f0 2024-03-25T17:30:10.810 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:10.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:10.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:10.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 10 monclient(hunting): my global_id is 4269 2024-03-25T17:30:10.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.809+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0300b28a0 con 0x7fc0300a98f0 2024-03-25T17:30:10.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.816+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000102037 0 0) 0x7fc0300b2a30 con 0x7fc0301430c0 2024-03-25T17:30:10.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.816+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 >> v1:172.21.15.96:6790/0 conn(0x7fc03013f710 legacy=0x7fc0301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:10.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 >> v1:172.21.15.96:6789/0 conn(0x7fc0300a98f0 legacy=0x7fc0300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:10.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:10.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 1 monclient: found mon.noname-c 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300b44c0 con 0x7fc0301430c0 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: _finish_auth 0 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc0280019f0 con 0x7fc0301430c0 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc027fff640 10 monclient: dump: 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:10.819 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc0376e4640 5 monclient: authenticate success, global_id 4264 2024-03-25T17:30:10.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.817+0000 7fc0376e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc028001b50 con 0x7fc0301430c0 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 1 -- 172.21.15.96:0/722738141 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc028001f60 con 0x7fc0301430c0 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc0376e4640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc0376e4640 4 set_mon_vals no callback set 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.828+0000 7fc027fff640 10 monclient: dump: 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.829 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:10.830 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:10.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 10 monclient: shutdown 2024-03-25T17:30:10.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:10.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 1 -- 172.21.15.96:0/722738141 >> v1:172.21.15.203:6789/0 conn(0x7fc0301430c0 legacy=0x7fc03014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:10.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 1 -- 172.21.15.96:0/722738141 shutdown_connections 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 1 -- 172.21.15.96:0/722738141 >> 172.21.15.96:0/722738141 conn(0x7fc0300b7ea0 msgr2=0x7fc0300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 1 -- 172.21.15.96:0/722738141 shutdown_connections 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.829+0000 7fc0376e4640 1 -- 172.21.15.96:0/722738141 wait complete. 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 10 monclient: build_initial_monmap 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 10 monclient: monmap: 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:10.833 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 librados: starting msgr at 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 librados: starting objecter 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 Processor -- start 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 -- start start 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 librados: setting wanted keys 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 1 librados: calling monclient init 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.830+0000 7fc0376e4640 10 monclient: init 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc0300a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc03013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:10.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc0301430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:10.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300ad460 con 0x7fc0301430c0 2024-03-25T17:30:10.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300b1670 con 0x7fc0300a98f0 2024-03-25T17:30:10.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0301d7a90 con 0x7fc03013f710 2024-03-25T17:30:10.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:10.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.831+0000 7fc0376e4640 10 monclient(hunting): authenticate will time out at 5369.113420s 2024-03-25T17:30:10.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.838+0000 7fc035480640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc0300a98f0 0x7fc0301d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:38118/0 (socket says 172.21.15.96:38118) 2024-03-25T17:30:10.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.838+0000 7fc035480640 1 -- 172.21.15.96:0/3492084122 learned_addr learned my addr 172.21.15.96:0/3492084122 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:10.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.839+0000 7fc025ffb640 10 client.?.objecter ms_handle_connect 0x7fc0300a98f0 2024-03-25T17:30:10.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.839+0000 7fc025ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:10.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.841+0000 7fc025ffb640 10 client.?.objecter ms_handle_connect 0x7fc0301430c0 2024-03-25T17:30:10.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.841+0000 7fc025ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:10.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.843+0000 7fc025ffb640 10 client.?.objecter ms_handle_connect 0x7fc03013f710 2024-03-25T17:30:10.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.843+0000 7fc025ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810165129 0 0) 0x7fc0300b44c0 con 0x7fc0301430c0 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T17:30:10.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.855+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014004eb0 con 0x7fc0301430c0 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271493786 0 0) 0x7fc014004eb0 con 0x7fc0301430c0 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0300b44c0 con 0x7fc0301430c0 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc028001dc0 con 0x7fc0301430c0 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:10.868 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:10.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.867+0000 7fc025ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:10.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2573878368 0 0) 0x7fc0300b44c0 con 0x7fc0301430c0 2024-03-25T17:30:10.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 >> v1:172.21.15.96:6790/0 conn(0x7fc03013f710 legacy=0x7fc0301d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:10.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 >> v1:172.21.15.203:6789/0 conn(0x7fc0300a98f0 legacy=0x7fc0301d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:10.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 1 monclient: found mon.a 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03014ece0 con 0x7fc0301430c0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 10 monclient: _finish_auth 0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc025ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 5 monclient: authenticate success, global_id 4272 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 10 monclient: _renew_subs 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03014dea0 con 0x7fc0301430c0 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 10 monclient: _renew_subs 2024-03-25T17:30:10.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0300b13e0 con 0x7fc0301430c0 2024-03-25T17:30:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.938+0000 7fc0376e4640 1 librados: init done 2024-03-25T17:30:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.939+0000 7fc0376e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.939+0000 7fc0376e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:10.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.939+0000 7fc0376e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:10.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:10.939+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0300627b0 con 0x7fc0301430c0 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc028001f20 con 0x7fc0301430c0 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc028003dc0 con 0x7fc0301430c0 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc025ffb640 10 monclient: dump: 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:11.193 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:11.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.192+0000 7fc0277fe640 4 set_mon_vals no callback set 2024-03-25T17:30:11.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.193+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fc028019d90 con 0x7fc0301430c0 2024-03-25T17:30:11.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.194+0000 7fc035480640 10 monclient: get_auth_request con 0x7fc01406dc20 auth_method 0 2024-03-25T17:30:11.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.194+0000 7fc025ffb640 10 client.4272.objecter ms_handle_connect 0x7fc01406dc20 2024-03-25T17:30:11.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fc0280538e0 con 0x7fc0301430c0 2024-03-25T17:30:11.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 10 client.4272.objecter ms_dispatch 0x7fc030000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:11.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:11.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:11.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T17:30:11.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc02804f3f0 con 0x7fc0301430c0 2024-03-25T17:30:11.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:11.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.199+0000 7fc025ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:11.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.424+0000 7fc0376e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:11.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.424+0000 7fc0376e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:11.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.424+0000 7fc0376e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:11.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.424+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc03013b430 con 0x7fc0301430c0 2024-03-25T17:30:11.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.449+0000 7fc025ffb640 1 -- 172.21.15.96:0/3492084122 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fc028051320 con 0x7fc0301430c0 2024-03-25T17:30:11.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.449+0000 7fc025ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:11.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.449+0000 7fc025ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:11.450 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:11.451 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.451+0000 7fc0376e4640 10 librados: watch_flush enter 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.451+0000 7fc0376e4640 10 librados: watch_flush exit 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.451+0000 7fc0376e4640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.451+0000 7fc0376e4640 10 client.4272.objecter successfully canceled tick 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 >> v1:172.21.15.203:6800/3611964061 conn(0x7fc01406dc20 legacy=0x7fc0140700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 10 monclient: shutdown 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 >> v1:172.21.15.96:6789/0 conn(0x7fc0301430c0 legacy=0x7fc03014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 shutdown_connections 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 >> 172.21.15.96:0/3492084122 conn(0x7fc0300b7ea0 msgr2=0x7fc030074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 shutdown_connections 2024-03-25T17:30:11.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 -- 172.21.15.96:0/3492084122 wait complete. 2024-03-25T17:30:11.454 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:11.452+0000 7fc0376e4640 1 librados: shutdown 2024-03-25T17:30:11.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:18.272 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:18.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.382+0000 7f2a5e97c640 10 monclient: get_monmap_and_config 2024-03-25T17:30:18.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.382+0000 7f2a5e97c640 10 monclient: build_initial_monmap 2024-03-25T17:30:18.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.382+0000 7f2a5e97c640 10 monclient: monmap: 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:18.384214+0000 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:18.384214+0000 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.383+0000 7f2a5e97c640 1 Processor -- start 2024-03-25T17:30:18.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.383+0000 7f2a5e97c640 1 -- start start 2024-03-25T17:30:18.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.383+0000 7f2a5e97c640 10 monclient: init 2024-03-25T17:30:18.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:18.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:18.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-c con 0x7f2a580a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:18.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:18.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a581430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a58142520 con 0x7f2a581430c0 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a580b28a0 con 0x7f2a580a98f0 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a580b2a30 con 0x7f2a5813f710 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.384+0000 7f2a5e97c640 10 monclient(hunting): authenticate will time out at 5376.666418s 2024-03-25T17:30:18.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.388+0000 7f2a577fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f2a5813f710 0x7f2a581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:35960/0 (socket says 172.21.15.96:35960) 2024-03-25T17:30:18.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.388+0000 7f2a577fe640 1 -- 172.21.15.96:0/2877646796 learned_addr learned my addr 172.21.15.96:0/2877646796 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.403+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810771543 0 0) 0x7f2a58142520 con 0x7f2a581430c0 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.403+0000 7f2a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.403+0000 7f2a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.403+0000 7f2a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.403+0000 7f2a56ffd640 10 monclient(hunting): my global_id is 4275 2024-03-25T17:30:18.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40003520 con 0x7f2a581430c0 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509367175 0 0) 0x7f2a580b2a30 con 0x7f2a5813f710 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 10 monclient(hunting): my global_id is 4271 2024-03-25T17:30:18.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.404+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a58142520 con 0x7f2a5813f710 2024-03-25T17:30:18.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237855282 0 0) 0x7f2a580b28a0 con 0x7f2a580a98f0 2024-03-25T17:30:18.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 10 monclient(hunting): my global_id is 4267 2024-03-25T17:30:18.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.424+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a580b2a30 con 0x7f2a580a98f0 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 128728378 0 0) 0x7f2a40003520 con 0x7f2a581430c0 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 >> v1:172.21.15.96:6790/0 conn(0x7f2a5813f710 legacy=0x7f2a581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 >> v1:172.21.15.203:6789/0 conn(0x7f2a580a98f0 legacy=0x7f2a580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 monclient: found mon.noname-a 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:18.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a580b44c0 con 0x7f2a581430c0 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: _finish_auth 0 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a5e97c640 5 monclient: authenticate success, global_id 4275 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2a50001a10 con 0x7f2a581430c0 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a5e97c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:18.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a56ffd640 10 monclient: dump: 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:18.430 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.425+0000 7f2a5e97c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:18.442 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a50001b70 con 0x7f2a581430c0 2024-03-25T17:30:18.442 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:18.442 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 1 -- 172.21.15.96:0/2877646796 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2a50001f80 con 0x7f2a581430c0 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a5e97c640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a5e97c640 4 set_mon_vals no callback set 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.441+0000 7f2a56ffd640 10 monclient: dump: 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.443 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:18.444 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 10 monclient: shutdown 2024-03-25T17:30:18.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 1 -- 172.21.15.96:0/2877646796 >> v1:172.21.15.96:6789/0 conn(0x7f2a581430c0 legacy=0x7f2a5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 1 -- 172.21.15.96:0/2877646796 shutdown_connections 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 1 -- 172.21.15.96:0/2877646796 >> 172.21.15.96:0/2877646796 conn(0x7f2a580b7ea0 msgr2=0x7f2a580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 1 -- 172.21.15.96:0/2877646796 shutdown_connections 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.442+0000 7f2a5e97c640 1 -- 172.21.15.96:0/2877646796 wait complete. 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 10 monclient: build_initial_monmap 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 10 monclient: monmap: 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:18.447 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 1 librados: starting msgr at 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 1 librados: starting objecter 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 1 Processor -- start 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 1 -- start start 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.443+0000 7f2a5e97c640 1 librados: setting wanted keys 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 1 librados: calling monclient init 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient: init 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:30:18.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-b con 0x7f2a580a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient(hunting): picked mon.noname-c con 0x7f2a581430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a580abb00 con 0x7f2a5813f710 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a580abde0 con 0x7f2a580a98f0 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a58138e20 con 0x7f2a581430c0 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:18.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.444+0000 7f2a5e97c640 10 monclient(hunting): authenticate will time out at 5376.726632s 2024-03-25T17:30:18.454 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.453+0000 7f2a5cf19640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f2a581430c0 0x7f2a580ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:36002/0 (socket says 172.21.15.96:36002) 2024-03-25T17:30:18.454 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.453+0000 7f2a5cf19640 1 -- 172.21.15.96:0/784690321 learned_addr learned my addr 172.21.15.96:0/784690321 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:18.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.455+0000 7f2a54ff9640 10 client.?.objecter ms_handle_connect 0x7f2a581430c0 2024-03-25T17:30:18.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.455+0000 7f2a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:18.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.455+0000 7f2a54ff9640 10 client.?.objecter ms_handle_connect 0x7f2a580a98f0 2024-03-25T17:30:18.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.455+0000 7f2a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:18.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786918804 0 0) 0x7f2a580abde0 con 0x7f2a580a98f0 2024-03-25T17:30:18.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 10 monclient(hunting): my global_id is 4270 2024-03-25T17:30:18.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.466+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a44004eb0 con 0x7f2a580a98f0 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676505470 0 0) 0x7f2a58138e20 con 0x7f2a581430c0 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 monclient(hunting): my global_id is 4274 2024-03-25T17:30:18.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a580abde0 con 0x7f2a581430c0 2024-03-25T17:30:18.472 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 client.?.objecter ms_handle_connect 0x7f2a5813f710 2024-03-25T17:30:18.472 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.470+0000 7f2a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:18.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3437852664 0 0) 0x7f2a44004eb0 con 0x7f2a580a98f0 2024-03-25T17:30:18.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a58138e20 con 0x7f2a580a98f0 2024-03-25T17:30:18.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2a48001970 con 0x7f2a580a98f0 2024-03-25T17:30:18.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:18.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 10 monclient(hunting): dump: 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:18.480 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:18.481 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.478+0000 7f2a54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:18.483 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.482+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567707570 0 0) 0x7f2a580abb00 con 0x7f2a5813f710 2024-03-25T17:30:18.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.482+0000 7f2a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:30:18.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.482+0000 7f2a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:18.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.482+0000 7f2a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:18.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.482+0000 7f2a54ff9640 10 monclient(hunting): my global_id is 4278 2024-03-25T17:30:18.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.483+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a44004eb0 con 0x7f2a5813f710 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3978233423 0 0) 0x7f2a580abde0 con 0x7f2a581430c0 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a580abb00 con 0x7f2a581430c0 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2a50001ec0 con 0x7f2a581430c0 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.486+0000 7f2a54ff9640 10 monclient(hunting): dump: 2024-03-25T17:30:18.487 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:18.488 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4021601510 0 0) 0x7f2a58138e20 con 0x7f2a580a98f0 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 >> v1:172.21.15.96:6790/0 conn(0x7f2a581430c0 legacy=0x7f2a580ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 >> v1:172.21.15.96:6789/0 conn(0x7f2a5813f710 legacy=0x7f2a58136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 1 monclient: found mon.b 2024-03-25T17:30:18.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a58137b00 con 0x7f2a580a98f0 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 10 monclient: _finish_auth 0 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 5 monclient: authenticate success, global_id 4270 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 10 monclient: _renew_subs 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a581e34b0 con 0x7f2a580a98f0 2024-03-25T17:30:18.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 10 monclient: _renew_subs 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a581e3820 con 0x7f2a580a98f0 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.487+0000 7f2a5e97c640 1 librados: init done 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.488+0000 7f2a5e97c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.488+0000 7f2a5e97c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.488+0000 7f2a5e97c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:18.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.488+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a580627b0 con 0x7f2a580a98f0 2024-03-25T17:30:18.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a48003610 con 0x7f2a580a98f0 2024-03-25T17:30:18.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:18.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2a48003920 con 0x7f2a580a98f0 2024-03-25T17:30:18.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:18.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a54ff9640 10 monclient: dump: 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:18.510 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:18.511 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:18.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.507+0000 7f2a567fc640 4 set_mon_vals no callback set 2024-03-25T17:30:18.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.508+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f2a48019930 con 0x7f2a580a98f0 2024-03-25T17:30:18.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.509+0000 7f2a577fe640 10 monclient: get_auth_request con 0x7f2a4406df70 auth_method 0 2024-03-25T17:30:18.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.510+0000 7f2a54ff9640 10 client.4270.objecter ms_handle_connect 0x7f2a4406df70 2024-03-25T17:30:18.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.511+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f2a48053690 con 0x7f2a580a98f0 2024-03-25T17:30:18.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.511+0000 7f2a54ff9640 10 client.4270.objecter ms_dispatch 0x7f2a58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:18.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.511+0000 7f2a54ff9640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:18.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.511+0000 7f2a54ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:18.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.511+0000 7f2a54ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T17:30:18.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.782+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a4804efd0 con 0x7f2a580a98f0 2024-03-25T17:30:18.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.782+0000 7f2a54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:18.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:18.782+0000 7f2a54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:19.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.003+0000 7f2a5e97c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:19.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.003+0000 7f2a5e97c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:19.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.003+0000 7f2a5e97c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:30:19.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.003+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a580b1760 con 0x7f2a580a98f0 2024-03-25T17:30:19.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.198+0000 7f2a54ff9640 1 -- 172.21.15.96:0/784690321 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f2a480510d0 con 0x7f2a580a98f0 2024-03-25T17:30:19.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.198+0000 7f2a54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:19.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.198+0000 7f2a54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:19.200 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:19.200 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 10 librados: watch_flush enter 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 10 librados: watch_flush exit 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 10 client.4270.objecter successfully canceled tick 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 >> v1:172.21.15.203:6800/3611964061 conn(0x7f2a4406df70 legacy=0x7f2a44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 10 monclient: shutdown 2024-03-25T17:30:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 >> v1:172.21.15.203:6789/0 conn(0x7f2a580a98f0 legacy=0x7f2a581385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 shutdown_connections 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 >> 172.21.15.96:0/784690321 conn(0x7f2a580b7ea0 msgr2=0x7f2a5814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 shutdown_connections 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 -- 172.21.15.96:0/784690321 wait complete. 2024-03-25T17:30:19.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:19.200+0000 7f2a5e97c640 1 librados: shutdown 2024-03-25T17:30:19.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:19.897 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:44.123 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.902 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:44.128 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.942 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:01:44.168 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.952 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:44.167 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.960 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:44.175 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.963 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:44.190 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.972 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:44.199 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:20.026 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:44.241 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:20.044 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:01:44.270 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:26.019 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:26.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.138+0000 7ff51bb0e640 10 monclient: get_monmap_and_config 2024-03-25T17:30:26.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.138+0000 7ff51bb0e640 10 monclient: build_initial_monmap 2024-03-25T17:30:26.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.138+0000 7ff51bb0e640 10 monclient: monmap: 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:26.139956+0000 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:26.139956+0000 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:26.140 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.139+0000 7ff51bb0e640 1 Processor -- start 2024-03-25T17:30:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.139+0000 7ff51bb0e640 1 -- start start 2024-03-25T17:30:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.139+0000 7ff51bb0e640 10 monclient: init 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7ff51406c380 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5140b12d0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7ff5140aedf0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b16f0 con 0x7ff51406c380 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140adfe0 con 0x7ff5140b12d0 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140add00 con 0x7ff5140aedf0 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:26.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.140+0000 7ff51bb0e640 10 monclient(hunting): authenticate will time out at 5384.422439s 2024-03-25T17:30:26.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.156+0000 7ff51a0ab640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ff5140aedf0 0x7ff51414e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:49320/0 (socket says 172.21.15.96:49320) 2024-03-25T17:30:26.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.156+0000 7ff51a0ab640 1 -- 172.21.15.96:0/2682942172 learned_addr learned my addr 172.21.15.96:0/2682942172 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469671799 0 0) 0x7ff5140b16f0 con 0x7ff51406c380 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 10 monclient(hunting): my global_id is 4281 2024-03-25T17:30:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.166+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500003520 con 0x7ff51406c380 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811637915 0 0) 0x7ff5140adfe0 con 0x7ff5140b12d0 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 10 monclient(hunting): my global_id is 4273 2024-03-25T17:30:26.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.170+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5140b16f0 con 0x7ff5140b12d0 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154223688 0 0) 0x7ff5140add00 con 0x7ff5140aedf0 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 10 monclient(hunting): my global_id is 4277 2024-03-25T17:30:26.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.176+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5140adfe0 con 0x7ff5140aedf0 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273106766 0 0) 0x7ff500003520 con 0x7ff51406c380 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 >> v1:172.21.15.96:6790/0 conn(0x7ff5140aedf0 legacy=0x7ff51414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 >> v1:172.21.15.203:6789/0 conn(0x7ff5140b12d0 legacy=0x7ff5140ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: _finish_hunting 0 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 monclient: found mon.noname-a 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140ae2c0 con 0x7ff51406c380 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: _finish_auth 0 2024-03-25T17:30:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: _check_auth_tickets 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff510001690 con 0x7ff51406c380 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff5188a8640 10 monclient: dump: 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.180 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff51bb0e640 5 monclient: authenticate success, global_id 4281 2024-03-25T17:30:26.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.178+0000 7ff51bb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:26.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5100017f0 con 0x7ff51406c380 2024-03-25T17:30:26.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:26.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 1 -- 172.21.15.96:0/2682942172 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff510001b80 con 0x7ff51406c380 2024-03-25T17:30:26.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff5188a8640 10 monclient: dump: 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff51bb0e640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:26.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.186+0000 7ff51bb0e640 4 set_mon_vals no callback set 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 10 monclient: shutdown 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 1 -- 172.21.15.96:0/2682942172 >> v1:172.21.15.96:6789/0 conn(0x7ff51406c380 legacy=0x7ff5140b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 1 -- 172.21.15.96:0/2682942172 shutdown_connections 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 1 -- 172.21.15.96:0/2682942172 >> 172.21.15.96:0/2682942172 conn(0x7ff5140b5d10 msgr2=0x7ff5140b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 1 -- 172.21.15.96:0/2682942172 shutdown_connections 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.187+0000 7ff51bb0e640 1 -- 172.21.15.96:0/2682942172 wait complete. 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.188+0000 7ff51bb0e640 10 monclient: build_initial_monmap 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.188+0000 7ff51bb0e640 10 monclient: monmap: 2024-03-25T17:30:26.190 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 librados: starting msgr at 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 librados: starting objecter 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 Processor -- start 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 -- start start 2024-03-25T17:30:26.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 librados: setting wanted keys 2024-03-25T17:30:26.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 1 librados: calling monclient init 2024-03-25T17:30:26.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.189+0000 7ff51bb0e640 10 monclient: init 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7ff51406c380 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5140aedf0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7ff5140b12d0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51406c8e0 con 0x7ff51406c380 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5141d79c0 con 0x7ff5140b12d0 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5141c4c00 con 0x7ff5140aedf0 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:26.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.190+0000 7ff51bb0e640 10 monclient(hunting): authenticate will time out at 5384.472293s 2024-03-25T17:30:26.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.202+0000 7ff5190a9640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ff5140aedf0 0x7ff5141d7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:49354/0 (socket says 172.21.15.96:49354) 2024-03-25T17:30:26.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.202+0000 7ff5190a9640 1 -- 172.21.15.96:0/3252886334 learned_addr learned my addr 172.21.15.96:0/3252886334 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:26.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.206+0000 7ff50a7fc640 10 client.?.objecter ms_handle_connect 0x7ff5140aedf0 2024-03-25T17:30:26.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.206+0000 7ff50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.210+0000 7ff50a7fc640 10 client.?.objecter ms_handle_connect 0x7ff5140b12d0 2024-03-25T17:30:26.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.210+0000 7ff50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:26.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.210+0000 7ff50a7fc640 10 client.?.objecter ms_handle_connect 0x7ff51406c380 2024-03-25T17:30:26.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.210+0000 7ff50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:26.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226382334 0 0) 0x7ff5141c4c00 con 0x7ff5140aedf0 2024-03-25T17:30:26.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T17:30:26.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.214+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f8004eb0 con 0x7ff5140aedf0 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995924445 0 0) 0x7ff51406c8e0 con 0x7ff51406c380 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T17:30:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.220+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5141c4c00 con 0x7ff51406c380 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107617441 0 0) 0x7ff4f8004eb0 con 0x7ff5140aedf0 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff51406c8e0 con 0x7ff5140aedf0 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff504001990 con 0x7ff5140aedf0 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 10 monclient(hunting): dump: 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.228 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.227+0000 7ff50a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688692272 0 0) 0x7ff5141d79c0 con 0x7ff5140b12d0 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:26.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:26.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T17:30:26.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.228+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f8004eb0 con 0x7ff5140b12d0 2024-03-25T17:30:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.239+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001879943 0 0) 0x7ff4f8004eb0 con 0x7ff5140b12d0 2024-03-25T17:30:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.239+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5141d79c0 con 0x7ff5140b12d0 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.239+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff50c001c70 con 0x7ff5140b12d0 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.239+0000 7ff50a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.240+0000 7ff50a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.240+0000 7ff50a7fc640 10 monclient(hunting): dump: 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.241 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982188621 0 0) 0x7ff5141c4c00 con 0x7ff51406c380 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f8004eb0 con 0x7ff51406c380 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff510001b00 con 0x7ff51406c380 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:26.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.241+0000 7ff50a7fc640 10 monclient(hunting): dump: 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.243 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3018782213 0 0) 0x7ff51406c8e0 con 0x7ff5140aedf0 2024-03-25T17:30:26.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 >> v1:172.21.15.203:6789/0 conn(0x7ff5140b12d0 legacy=0x7ff5141e23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 >> v1:172.21.15.96:6789/0 conn(0x7ff51406c380 legacy=0x7ff5141c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 1 monclient: found mon.c 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5141c4ee0 con 0x7ff5140aedf0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 5 monclient: authenticate success, global_id 4280 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 10 monclient: _renew_subs 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5141c57f0 con 0x7ff5140aedf0 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:26.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 10 monclient: _renew_subs 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5141d8740 con 0x7ff5140aedf0 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.244+0000 7ff51bb0e640 1 librados: init done 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.245+0000 7ff51bb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.245+0000 7ff51bb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.245+0000 7ff51bb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:26.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.245+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5140627b0 con 0x7ff5140aedf0 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5040035f0 con 0x7ff5140aedf0 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff5040039c0 con 0x7ff5140aedf0 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:26.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50a7fc640 10 monclient: dump: 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:26.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.546+0000 7ff50bfff640 4 set_mon_vals no callback set 2024-03-25T17:30:26.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.547+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ff504003be0 con 0x7ff5140aedf0 2024-03-25T17:30:26.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.547+0000 7ff5198aa640 10 monclient: get_auth_request con 0x7ff4f806e160 auth_method 0 2024-03-25T17:30:26.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.547+0000 7ff50a7fc640 10 client.4280.objecter ms_handle_connect 0x7ff4f806e160 2024-03-25T17:30:26.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.554+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ff504054770 con 0x7ff5140aedf0 2024-03-25T17:30:26.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.554+0000 7ff50a7fc640 10 client.4280.objecter ms_dispatch 0x7ff514000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:26.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.554+0000 7ff50a7fc640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:26.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.554+0000 7ff50a7fc640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:26.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.554+0000 7ff50a7fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T17:30:26.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.555+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff504050280 con 0x7ff5140aedf0 2024-03-25T17:30:26.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.555+0000 7ff50a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:26.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.555+0000 7ff50a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:26.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.803+0000 7ff51bb0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:26.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.803+0000 7ff51bb0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:26.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.803+0000 7ff51bb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:26.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.803+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5140ae980 con 0x7ff5140aedf0 2024-03-25T17:30:26.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.985+0000 7ff50a7fc640 1 -- 172.21.15.96:0/3252886334 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ff5040521b0 con 0x7ff5140aedf0 2024-03-25T17:30:26.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.985+0000 7ff50a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:26.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.985+0000 7ff50a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:26.986 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:26.986 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:26.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 10 librados: watch_flush enter 2024-03-25T17:30:26.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 10 librados: watch_flush exit 2024-03-25T17:30:26.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T17:30:26.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 10 client.4280.objecter successfully canceled tick 2024-03-25T17:30:26.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 >> v1:172.21.15.203:6800/3611964061 conn(0x7ff4f806e160 legacy=0x7ff4f8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 10 monclient: shutdown 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 >> v1:172.21.15.96:6790/0 conn(0x7ff5140aedf0 legacy=0x7ff5141d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 shutdown_connections 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 >> 172.21.15.96:0/3252886334 conn(0x7ff5140b5d10 msgr2=0x7ff5140b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 shutdown_connections 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 -- 172.21.15.96:0/3252886334 wait complete. 2024-03-25T17:30:26.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:26.987+0000 7ff51bb0e640 1 librados: shutdown 2024-03-25T17:30:27.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:33.805 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:33.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.923+0000 7f25ccdf4640 10 monclient: get_monmap_and_config 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.923+0000 7f25ccdf4640 10 monclient: build_initial_monmap 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.923+0000 7f25ccdf4640 10 monclient: monmap: 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:33.925227+0000 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:33.925227+0000 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:33.925 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:33.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.924+0000 7f25ccdf4640 1 Processor -- start 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.924+0000 7f25ccdf4640 1 -- start start 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.924+0000 7f25ccdf4640 10 monclient: init 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-b con 0x7f25c80a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-a con 0x7f25c813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f25c81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c8142520 con 0x7f25c813f710 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80b28a0 con 0x7f25c81430c0 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80b2a30 con 0x7f25c80a98f0 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:33.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.925+0000 7f25ccdf4640 10 monclient(hunting): authenticate will time out at 5392.207571s 2024-03-25T17:30:33.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.931+0000 7f25c6d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f25c81430c0 0x7f25c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:57440/0 (socket says 172.21.15.96:57440) 2024-03-25T17:30:33.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.931+0000 7f25c6d9d640 1 -- 172.21.15.96:0/3430491775 learned_addr learned my addr 172.21.15.96:0/3430491775 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147689609 0 0) 0x7f25c80b28a0 con 0x7f25c81430c0 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 10 monclient(hunting): my global_id is 4279 2024-03-25T17:30:33.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.955+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b4003520 con 0x7f25c81430c0 2024-03-25T17:30:33.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500702103 0 0) 0x7f25c8142520 con 0x7f25c813f710 2024-03-25T17:30:33.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 10 monclient(hunting): my global_id is 4287 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.957+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c80b28a0 con 0x7f25c813f710 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644988 0 0) 0x7f25c80b2a30 con 0x7f25c80a98f0 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 10 monclient(hunting): my global_id is 4283 2024-03-25T17:30:33.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.958+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8142520 con 0x7f25c80a98f0 2024-03-25T17:30:33.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248229207 0 0) 0x7f25c8142520 con 0x7f25c80a98f0 2024-03-25T17:30:33.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 >> v1:172.21.15.203:6789/0 conn(0x7f25c81430c0 legacy=0x7f25c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:33.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 >> v1:172.21.15.96:6789/0 conn(0x7f25c813f710 legacy=0x7f25c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:33.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:30:33.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 monclient: found mon.noname-b 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80b44c0 con 0x7f25c80a98f0 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: _finish_auth 0 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f25b80015b0 con 0x7f25c80a98f0 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25c5d9b640 10 monclient: dump: 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:33.979 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25ccdf4640 5 monclient: authenticate success, global_id 4283 2024-03-25T17:30:33.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.977+0000 7f25ccdf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b8001710 con 0x7f25c80a98f0 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 1 -- 172.21.15.96:0/3430491775 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f25b8001b60 con 0x7f25c80a98f0 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25c5d9b640 10 monclient: dump: 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:33.991 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25ccdf4640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:33.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.990+0000 7f25ccdf4640 4 set_mon_vals no callback set 2024-03-25T17:30:33.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 10 monclient: shutdown 2024-03-25T17:30:33.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:33.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 1 -- 172.21.15.96:0/3430491775 >> v1:172.21.15.96:6790/0 conn(0x7f25c80a98f0 legacy=0x7f25c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 1 -- 172.21.15.96:0/3430491775 shutdown_connections 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 1 -- 172.21.15.96:0/3430491775 >> 172.21.15.96:0/3430491775 conn(0x7f25c80b7ea0 msgr2=0x7f25c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 1 -- 172.21.15.96:0/3430491775 shutdown_connections 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.991+0000 7f25ccdf4640 1 -- 172.21.15.96:0/3430491775 wait complete. 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.992+0000 7f25ccdf4640 10 monclient: build_initial_monmap 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.992+0000 7f25ccdf4640 10 monclient: monmap: 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:33.994 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.992+0000 7f25ccdf4640 1 librados: starting msgr at 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.992+0000 7f25ccdf4640 1 librados: starting objecter 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 1 Processor -- start 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 1 -- start start 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 1 librados: setting wanted keys 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 1 librados: calling monclient init 2024-03-25T17:30:33.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient: init 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-a con 0x7f25c80a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f25c813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 10 monclient(hunting): picked mon.noname-b con 0x7f25c81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:33.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.993+0000 7f25ccdf4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c8138050 con 0x7f25c80a98f0 2024-03-25T17:30:33.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.994+0000 7f25ccdf4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c8137d70 con 0x7f25c81430c0 2024-03-25T17:30:33.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.994+0000 7f25ccdf4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c8138f20 con 0x7f25c813f710 2024-03-25T17:30:33.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.994+0000 7f25ccdf4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:33.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:33.994+0000 7f25ccdf4640 10 monclient(hunting): authenticate will time out at 5392.276015s 2024-03-25T17:30:34.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.003+0000 7f25c6d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f25c81430c0 0x7f25c81e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:57504/0 (socket says 172.21.15.96:57504) 2024-03-25T17:30:34.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.003+0000 7f25c6d9d640 1 -- 172.21.15.96:0/2915878899 learned_addr learned my addr 172.21.15.96:0/2915878899 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:34.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.005+0000 7f25beffd640 10 client.?.objecter ms_handle_connect 0x7f25c81430c0 2024-03-25T17:30:34.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.005+0000 7f25beffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:34.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.012+0000 7f25beffd640 10 client.?.objecter ms_handle_connect 0x7f25c80a98f0 2024-03-25T17:30:34.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.012+0000 7f25beffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909258543 0 0) 0x7f25c8137d70 con 0x7f25c81430c0 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T17:30:34.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.015+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8004eb0 con 0x7f25c81430c0 2024-03-25T17:30:34.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568385598 0 0) 0x7f25c8138050 con 0x7f25c80a98f0 2024-03-25T17:30:34.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:34.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:34.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:34.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T17:30:34.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.022+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8137d70 con 0x7f25c80a98f0 2024-03-25T17:30:34.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.026+0000 7f25beffd640 10 client.?.objecter ms_handle_connect 0x7f25c813f710 2024-03-25T17:30:34.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.026+0000 7f25beffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 503305914 0 0) 0x7f25c8137d70 con 0x7f25c80a98f0 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c8138050 con 0x7f25c80a98f0 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f25b8001a20 con 0x7f25c80a98f0 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 10 monclient(hunting): dump: 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:34.035 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:34.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.034+0000 7f25beffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:34.043 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815882184 0 0) 0x7f25a8004eb0 con 0x7f25c81430c0 2024-03-25T17:30:34.043 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c8137d70 con 0x7f25c81430c0 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f25c0001b70 con 0x7f25c81430c0 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.042+0000 7f25beffd640 10 monclient(hunting): dump: 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:34.044 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.045+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3132084130 0 0) 0x7f25c8138050 con 0x7f25c80a98f0 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 >> v1:172.21.15.96:6790/0 conn(0x7f25c813f710 legacy=0x7f25c8138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 >> v1:172.21.15.203:6789/0 conn(0x7f25c81430c0 legacy=0x7f25c81e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 10 monclient: _finish_hunting 0 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 1 monclient: found mon.a 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c81379b0 con 0x7f25c80a98f0 2024-03-25T17:30:34.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 10 monclient: _finish_auth 0 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 10 monclient: _check_auth_tickets 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 5 monclient: authenticate success, global_id 4290 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 10 monclient: _renew_subs 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c8138960 con 0x7f25c80a98f0 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 10 monclient: _renew_subs 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c81e32d0 con 0x7f25c80a98f0 2024-03-25T17:30:34.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.046+0000 7f25ccdf4640 1 librados: init done 2024-03-25T17:30:34.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.047+0000 7f25ccdf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:34.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.047+0000 7f25ccdf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:34.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.047+0000 7f25ccdf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:34.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.047+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c8137ba0 con 0x7f25c80a98f0 2024-03-25T17:30:34.348 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.347+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b80038a0 con 0x7f25c80a98f0 2024-03-25T17:30:34.348 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f25b8001c20 con 0x7f25c80a98f0 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 10 monclient: dump: 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:34.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25c559a640 4 set_mon_vals no callback set 2024-03-25T17:30:34.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.348+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f25b8019a20 con 0x7f25c80a98f0 2024-03-25T17:30:34.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.349+0000 7f25bffff640 10 monclient: get_auth_request con 0x7f25a806de90 auth_method 0 2024-03-25T17:30:34.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.349+0000 7f25beffd640 10 client.4290.objecter ms_handle_connect 0x7f25a806de90 2024-03-25T17:30:34.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f25b8053660 con 0x7f25c80a98f0 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 10 client.4290.objecter ms_dispatch 0x7f25c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25b804f170 con 0x7f25c80a98f0 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:34.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.357+0000 7f25beffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:34.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.606+0000 7f25ccdf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:34.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.606+0000 7f25ccdf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:34.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.606+0000 7f25ccdf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:34.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.606+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c80627b0 con 0x7f25c80a98f0 2024-03-25T17:30:34.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.631+0000 7f25beffd640 1 -- 172.21.15.96:0/2915878899 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f25b80510a0 con 0x7f25c80a98f0 2024-03-25T17:30:34.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.631+0000 7f25beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:34.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.631+0000 7f25beffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:34.632 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:34.632 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:34.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 10 librados: watch_flush enter 2024-03-25T17:30:34.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 10 librados: watch_flush exit 2024-03-25T17:30:34.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 10 client.4290.objecter successfully canceled tick 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 >> v1:172.21.15.203:6800/3611964061 conn(0x7f25a806de90 legacy=0x7f25a8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 10 monclient: shutdown 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 >> v1:172.21.15.96:6789/0 conn(0x7f25c80a98f0 legacy=0x7f25c81db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 shutdown_connections 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 >> 172.21.15.96:0/2915878899 conn(0x7f25c80b7ea0 msgr2=0x7f25c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 shutdown_connections 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 -- 172.21.15.96:0/2915878899 wait complete. 2024-03-25T17:30:34.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:34.633+0000 7f25ccdf4640 1 librados: shutdown 2024-03-25T17:30:34.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:40.034 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:04.260 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.042 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:04.268 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.071 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:04.298 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.096 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:04.322 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.106 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:04.332 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.109 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:04.324 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.114 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:04.329 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.140 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:04.366 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:40.145 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:04.360 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:41.453 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.568+0000 7fde774d3640 10 monclient: get_monmap_and_config 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.568+0000 7fde774d3640 10 monclient: build_initial_monmap 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.568+0000 7fde774d3640 10 monclient: monmap: 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:41.568 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:41.569080+0000 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:41.569080+0000 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.569+0000 7fde774d3640 1 Processor -- start 2024-03-25T17:30:41.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.569+0000 7fde774d3640 1 -- start start 2024-03-25T17:30:41.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.569+0000 7fde774d3640 10 monclient: init 2024-03-25T17:30:41.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:41.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-c con 0x7fde7013fc60 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-b con 0x7fde70070570 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-a con 0x7fde700727a0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7013c9f0 con 0x7fde700727a0 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7006fac0 con 0x7fde7013fc60 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700715d0 con 0x7fde70070570 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:41.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.570+0000 7fde774d3640 10 monclient(hunting): authenticate will time out at 5399.851858s 2024-03-25T17:30:41.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.584+0000 7fde7526f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fde7013fc60 0x7fde7006f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:44104/0 (socket says 172.21.15.96:44104) 2024-03-25T17:30:41.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.584+0000 7fde7526f640 1 -- 172.21.15.96:0/2787850948 learned_addr learned my addr 172.21.15.96:0/2787850948 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749738756 0 0) 0x7fde700715d0 con 0x7fde70070570 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T17:30:41.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.594+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c003520 con 0x7fde70070570 2024-03-25T17:30:41.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275291760 0 0) 0x7fde7006fac0 con 0x7fde7013fc60 2024-03-25T17:30:41.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 10 monclient(hunting): my global_id is 4285 2024-03-25T17:30:41.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.596+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde700715d0 con 0x7fde7013fc60 2024-03-25T17:30:41.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570663958 0 0) 0x7fde7013c9f0 con 0x7fde700727a0 2024-03-25T17:30:41.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient(hunting): my global_id is 4293 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7006fac0 con 0x7fde700727a0 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1004867035 0 0) 0x7fde700715d0 con 0x7fde7013fc60 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 >> v1:172.21.15.96:6790/0 conn(0x7fde70070570 legacy=0x7fde70072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 >> v1:172.21.15.96:6789/0 conn(0x7fde700727a0 legacy=0x7fde7014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 monclient: found mon.noname-c 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700718b0 con 0x7fde7013fc60 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: _finish_auth 0 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fde6c0015f0 con 0x7fde7013fc60 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:41.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde67fff640 10 monclient: dump: 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde774d3640 5 monclient: authenticate success, global_id 4285 2024-03-25T17:30:41.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.606+0000 7fde774d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:41.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde6c001750 con 0x7fde7013fc60 2024-03-25T17:30:41.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:41.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 1 -- 172.21.15.96:0/2787850948 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fde6c001b60 con 0x7fde7013fc60 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde67fff640 10 monclient: dump: 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde774d3640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:41.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde774d3640 4 set_mon_vals no callback set 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde774d3640 10 monclient: shutdown 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde774d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.623+0000 7fde774d3640 1 -- 172.21.15.96:0/2787850948 >> v1:172.21.15.203:6789/0 conn(0x7fde7013fc60 legacy=0x7fde7006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 1 -- 172.21.15.96:0/2787850948 shutdown_connections 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 1 -- 172.21.15.96:0/2787850948 >> 172.21.15.96:0/2787850948 conn(0x7fde700b4a80 msgr2=0x7fde700b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 1 -- 172.21.15.96:0/2787850948 shutdown_connections 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 1 -- 172.21.15.96:0/2787850948 wait complete. 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 10 monclient: build_initial_monmap 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.624+0000 7fde774d3640 10 monclient: monmap: 2024-03-25T17:30:41.625 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 librados: starting msgr at 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 librados: starting objecter 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 Processor -- start 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 -- start start 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 librados: setting wanted keys 2024-03-25T17:30:41.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 1 librados: calling monclient init 2024-03-25T17:30:41.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.625+0000 7fde774d3640 10 monclient: init 2024-03-25T17:30:41.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:41.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-a con 0x7fde70070570 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-c con 0x7fde700727a0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient(hunting): picked mon.noname-b con 0x7fde7013fc60 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70137ba0 con 0x7fde70070570 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde701d4220 con 0x7fde7013fc60 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde701d7720 con 0x7fde700727a0 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:41.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.626+0000 7fde774d3640 10 monclient(hunting): authenticate will time out at 5399.907556s 2024-03-25T17:30:41.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.643+0000 7fde74a6e640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fde700727a0 0x7fde701d4720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:39604/0 (socket says 172.21.15.96:39604) 2024-03-25T17:30:41.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.643+0000 7fde74a6e640 1 -- 172.21.15.96:0/465562251 learned_addr learned my addr 172.21.15.96:0/465562251 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:41.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.649+0000 7fde65ffb640 10 client.?.objecter ms_handle_connect 0x7fde70070570 2024-03-25T17:30:41.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.649+0000 7fde65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:41.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.649+0000 7fde65ffb640 10 client.?.objecter ms_handle_connect 0x7fde700727a0 2024-03-25T17:30:41.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.649+0000 7fde65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:41.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.652+0000 7fde65ffb640 10 client.?.objecter ms_handle_connect 0x7fde7013fc60 2024-03-25T17:30:41.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.652+0000 7fde65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:41.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715840902 0 0) 0x7fde701d4220 con 0x7fde7013fc60 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde7013fc60 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117868189 0 0) 0x7fde70137ba0 con 0x7fde70070570 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 10 monclient(hunting): my global_id is 4296 2024-03-25T17:30:41.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.662+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde701d4220 con 0x7fde70070570 2024-03-25T17:30:41.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.674+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112008132 0 0) 0x7fde701d4220 con 0x7fde70070570 2024-03-25T17:30:41.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.674+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde70137ba0 con 0x7fde70070570 2024-03-25T17:30:41.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.675+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fde6c001aa0 con 0x7fde70070570 2024-03-25T17:30:41.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.675+0000 7fde65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:41.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.675+0000 7fde65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:41.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.675+0000 7fde65ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:41.674 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.675+0000 7fde65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215931756 0 0) 0x7fde701d7720 con 0x7fde700727a0 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T17:30:41.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.676+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde701d4220 con 0x7fde700727a0 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015195877 0 0) 0x7fde54004eb0 con 0x7fde7013fc60 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde701d7720 con 0x7fde7013fc60 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fde68001bf0 con 0x7fde7013fc60 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.689+0000 7fde65ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:41.689 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:41.690 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:41.690 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:41.690 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:41.690 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:41.690 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3125196015 0 0) 0x7fde70137ba0 con 0x7fde70070570 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 >> v1:172.21.15.96:6790/0 conn(0x7fde700727a0 legacy=0x7fde701d4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 >> v1:172.21.15.203:6789/0 conn(0x7fde7013fc60 legacy=0x7fde701d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 1 monclient: found mon.a 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70141050 con 0x7fde70070570 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 10 monclient: _finish_auth 0 2024-03-25T17:30:41.691 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 5 monclient: authenticate success, global_id 4296 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 10 monclient: _renew_subs 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde701422e0 con 0x7fde70070570 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 10 monclient: _renew_subs 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde701d5010 con 0x7fde70070570 2024-03-25T17:30:41.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.691+0000 7fde774d3640 1 librados: init done 2024-03-25T17:30:41.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.692+0000 7fde774d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:41.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.692+0000 7fde774d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:41.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.692+0000 7fde774d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:41.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:41.692+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70141240 con 0x7fde70070570 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde6c004a90 con 0x7fde70070570 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fde6c004da0 con 0x7fde70070570 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde65ffb640 10 monclient: dump: 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:42.017 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:42.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.017+0000 7fde677fe640 4 set_mon_vals no callback set 2024-03-25T17:30:42.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.018+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fde6c01ad70 con 0x7fde70070570 2024-03-25T17:30:42.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.019+0000 7fde74a6e640 10 monclient: get_auth_request con 0x7fde5406dec0 auth_method 0 2024-03-25T17:30:42.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.019+0000 7fde65ffb640 10 client.4296.objecter ms_handle_connect 0x7fde5406dec0 2024-03-25T17:30:42.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fde6c0548c0 con 0x7fde70070570 2024-03-25T17:30:42.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 10 client.4296.objecter ms_dispatch 0x7fde70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:42.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:42.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:42.024 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T17:30:42.024 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde6c0503d0 con 0x7fde70070570 2024-03-25T17:30:42.024 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:42.024 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.023+0000 7fde65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:42.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.272+0000 7fde774d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:42.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.272+0000 7fde774d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:42.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.272+0000 7fde774d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:42.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.272+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde701d4410 con 0x7fde70070570 2024-03-25T17:30:42.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.301+0000 7fde65ffb640 1 -- 172.21.15.96:0/465562251 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fde6c052300 con 0x7fde70070570 2024-03-25T17:30:42.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.301+0000 7fde65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:42.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.301+0000 7fde65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:42.301 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:42.301 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.303+0000 7fde774d3640 10 librados: watch_flush enter 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.303+0000 7fde774d3640 10 librados: watch_flush exit 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.303+0000 7fde774d3640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.303+0000 7fde774d3640 10 client.4296.objecter successfully canceled tick 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 >> v1:172.21.15.203:6800/3611964061 conn(0x7fde5406dec0 legacy=0x7fde54070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 10 monclient: shutdown 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 >> v1:172.21.15.96:6789/0 conn(0x7fde70070570 legacy=0x7fde70141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:42.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 shutdown_connections 2024-03-25T17:30:42.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 >> 172.21.15.96:0/465562251 conn(0x7fde700b4a80 msgr2=0x7fde70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:42.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 shutdown_connections 2024-03-25T17:30:42.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 -- 172.21.15.96:0/465562251 wait complete. 2024-03-25T17:30:42.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:42.304+0000 7fde774d3640 1 librados: shutdown 2024-03-25T17:30:42.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:49.122 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.237+0000 7fe87900f640 10 monclient: get_monmap_and_config 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.237+0000 7fe87900f640 10 monclient: build_initial_monmap 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.237+0000 7fe87900f640 10 monclient: monmap: 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:49.238732+0000 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:49.238732+0000 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:49.238 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:49.239 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 1 Processor -- start 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 1 -- start start 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient: init 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-b con 0x7fe8740a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-a con 0x7fe87413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-c con 0x7fe8741430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe874142520 con 0x7fe87413f710 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740b28a0 con 0x7fe8741430c0 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.238+0000 7fe87900f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740b2a30 con 0x7fe8740a98f0 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.239+0000 7fe87900f640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:49.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.239+0000 7fe87900f640 10 monclient(hunting): authenticate will time out at 5407.520953s 2024-03-25T17:30:49.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.249+0000 7fe87359e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe8741430c0 0x7fe87414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:35858/0 (socket says 172.21.15.96:35858) 2024-03-25T17:30:49.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.249+0000 7fe87359e640 1 -- 172.21.15.96:0/411404669 learned_addr learned my addr 172.21.15.96:0/411404669 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:49.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.261+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176066922 0 0) 0x7fe8740b28a0 con 0x7fe8741430c0 2024-03-25T17:30:49.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.261+0000 7fe871d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.261+0000 7fe871d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.263 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.261+0000 7fe871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.263 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.261+0000 7fe871d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T17:30:49.263 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.262+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe85c003520 con 0x7fe8741430c0 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295989851 0 0) 0x7fe8740b2a30 con 0x7fe8740a98f0 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 10 monclient(hunting): my global_id is 4295 2024-03-25T17:30:49.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.268+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8740b28a0 con 0x7fe8740a98f0 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585345575 0 0) 0x7fe874142520 con 0x7fe87413f710 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 10 monclient(hunting): my global_id is 4299 2024-03-25T17:30:49.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.271+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8740b2a30 con 0x7fe87413f710 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619722914 0 0) 0x7fe8740b2a30 con 0x7fe87413f710 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 >> v1:172.21.15.96:6790/0 conn(0x7fe8740a98f0 legacy=0x7fe8740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 >> v1:172.21.15.203:6789/0 conn(0x7fe8741430c0 legacy=0x7fe87414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 monclient: found mon.noname-a 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740b44c0 con 0x7fe87413f710 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: _finish_auth 0 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:30:49.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe8680015b0 con 0x7fe87413f710 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe87900f640 5 monclient: authenticate success, global_id 4299 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe87900f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe871d9b640 10 monclient: dump: 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:49.286 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:49.287 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.284+0000 7fe87900f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe868001710 con 0x7fe87413f710 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 1 -- 172.21.15.96:0/411404669 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe868001b20 con 0x7fe87413f710 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe87900f640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe87900f640 4 set_mon_vals no callback set 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:49.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.299+0000 7fe871d9b640 10 monclient: dump: 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:49.301 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 10 monclient: shutdown 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 1 -- 172.21.15.96:0/411404669 >> v1:172.21.15.96:6789/0 conn(0x7fe87413f710 legacy=0x7fe8741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 1 -- 172.21.15.96:0/411404669 shutdown_connections 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 1 -- 172.21.15.96:0/411404669 >> 172.21.15.96:0/411404669 conn(0x7fe8740b7ea0 msgr2=0x7fe8740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 1 -- 172.21.15.96:0/411404669 shutdown_connections 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.300+0000 7fe87900f640 1 -- 172.21.15.96:0/411404669 wait complete. 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.301+0000 7fe87900f640 10 monclient: build_initial_monmap 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.301+0000 7fe87900f640 10 monclient: monmap: 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:49.303 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 librados: starting msgr at 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 librados: starting objecter 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 Processor -- start 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 -- start start 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 librados: setting wanted keys 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 1 librados: calling monclient init 2024-03-25T17:30:49.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.302+0000 7fe87900f640 10 monclient: init 2024-03-25T17:30:49.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:49.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:30:49.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-c con 0x7fe8740a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-b con 0x7fe87413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient(hunting): picked mon.noname-a con 0x7fe8741430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87413a570 con 0x7fe8741430c0 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87413a0e0 con 0x7fe87413f710 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740b7c60 con 0x7fe8740a98f0 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:49.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.303+0000 7fe87900f640 10 monclient(hunting): authenticate will time out at 5407.585477s 2024-03-25T17:30:49.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.314+0000 7fe872d9d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fe8740a98f0 0x7fe8740b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:37870/0 (socket says 172.21.15.96:37870) 2024-03-25T17:30:49.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.314+0000 7fe872d9d640 1 -- 172.21.15.96:0/4115078566 learned_addr learned my addr 172.21.15.96:0/4115078566 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:49.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.316+0000 7fe85b7fe640 10 client.?.objecter ms_handle_connect 0x7fe8740a98f0 2024-03-25T17:30:49.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.316+0000 7fe85b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:49.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.320+0000 7fe85b7fe640 10 client.?.objecter ms_handle_connect 0x7fe8741430c0 2024-03-25T17:30:49.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.320+0000 7fe85b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242896749 0 0) 0x7fe8740b7c60 con 0x7fe8740a98f0 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 10 monclient(hunting): my global_id is 4298 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.322+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe860004eb0 con 0x7fe8740a98f0 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.323+0000 7fe85b7fe640 10 client.?.objecter ms_handle_connect 0x7fe87413f710 2024-03-25T17:30:49.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.323+0000 7fe85b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:49.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821270711 0 0) 0x7fe87413a0e0 con 0x7fe87413f710 2024-03-25T17:30:49.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T17:30:49.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.328+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8740b7c60 con 0x7fe87413f710 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845782100 0 0) 0x7fe87413a570 con 0x7fe8741430c0 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 10 monclient(hunting): my global_id is 4302 2024-03-25T17:30:49.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.330+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87413a0e0 con 0x7fe8741430c0 2024-03-25T17:30:49.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.334+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956624974 0 0) 0x7fe860004eb0 con 0x7fe8740a98f0 2024-03-25T17:30:49.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.334+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe87413a570 con 0x7fe8740a98f0 2024-03-25T17:30:49.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.334+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe864001a10 con 0x7fe8740a98f0 2024-03-25T17:30:49.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.334+0000 7fe85b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:49.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.335+0000 7fe85b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.335+0000 7fe85b7fe640 10 monclient(hunting): dump: 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.335+0000 7fe85b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159629891 0 0) 0x7fe8740b7c60 con 0x7fe87413f710 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe860004eb0 con 0x7fe87413f710 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe868001a20 con 0x7fe87413f710 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:49.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.337+0000 7fe85b7fe640 10 monclient(hunting): dump: 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:49.339 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 278533137 0 0) 0x7fe87413a570 con 0x7fe8740a98f0 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 >> v1:172.21.15.203:6789/0 conn(0x7fe87413f710 legacy=0x7fe87412d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 >> v1:172.21.15.96:6789/0 conn(0x7fe8741430c0 legacy=0x7fe8741d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 1 monclient: found mon.c 2024-03-25T17:30:49.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe874139d20 con 0x7fe8740a98f0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe85b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 5 monclient: authenticate success, global_id 4298 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 10 monclient: _renew_subs 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87412d9e0 con 0x7fe8740a98f0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 10 monclient: _renew_subs 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87412f650 con 0x7fe8740a98f0 2024-03-25T17:30:49.345 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.343+0000 7fe87900f640 1 librados: init done 2024-03-25T17:30:49.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.344+0000 7fe87900f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:49.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.344+0000 7fe87900f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:49.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.344+0000 7fe87900f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:49.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.344+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740b3ef0 con 0x7fe8740a98f0 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe864001c50 con 0x7fe8740a98f0 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe864003a60 con 0x7fe8740a98f0 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe85b7fe640 10 monclient: dump: 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.640 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.639+0000 7fe87159a640 4 set_mon_vals no callback set 2024-03-25T17:30:49.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.640+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fe864003c80 con 0x7fe8740a98f0 2024-03-25T17:30:49.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.640+0000 7fe87259c640 10 monclient: get_auth_request con 0x7fe86006df70 auth_method 0 2024-03-25T17:30:49.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.640+0000 7fe85b7fe640 10 client.4298.objecter ms_handle_connect 0x7fe86006df70 2024-03-25T17:30:49.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fe864054890 con 0x7fe8740a98f0 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 10 client.4298.objecter ms_dispatch 0x7fe874000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8640503a0 con 0x7fe8740a98f0 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:49.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.645+0000 7fe85b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:49.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.888+0000 7fe87900f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:49.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.888+0000 7fe87900f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:49.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.888+0000 7fe87900f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:30:49.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.888+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8740627b0 con 0x7fe8740a98f0 2024-03-25T17:30:49.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.962+0000 7fe85b7fe640 1 -- 172.21.15.96:0/4115078566 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fe8640522d0 con 0x7fe8740a98f0 2024-03-25T17:30:49.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.962+0000 7fe85b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:49.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.962+0000 7fe85b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:49.963 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:49.963 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.964+0000 7fe87900f640 10 librados: watch_flush enter 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.964+0000 7fe87900f640 10 librados: watch_flush exit 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.964+0000 7fe87900f640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.964+0000 7fe87900f640 10 client.4298.objecter successfully canceled tick 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.964+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 >> v1:172.21.15.203:6800/3611964061 conn(0x7fe86006df70 legacy=0x7fe860070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 10 monclient: shutdown 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:49.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 >> v1:172.21.15.96:6790/0 conn(0x7fe8740a98f0 legacy=0x7fe8740b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:49.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 shutdown_connections 2024-03-25T17:30:49.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 >> 172.21.15.96:0/4115078566 conn(0x7fe8740b7ea0 msgr2=0x7fe874074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:49.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 shutdown_connections 2024-03-25T17:30:49.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 -- 172.21.15.96:0/4115078566 wait complete. 2024-03-25T17:30:49.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:49.965+0000 7fe87900f640 1 librados: shutdown 2024-03-25T17:30:49.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:56.786 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:56.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.885+0000 7feeb2315640 10 monclient: get_monmap_and_config 2024-03-25T17:30:56.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.885+0000 7feeb2315640 10 monclient: build_initial_monmap 2024-03-25T17:30:56.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.885+0000 7feeb2315640 10 monclient: monmap: 2024-03-25T17:30:56.886 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:56.886 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:30:56.886936+0000 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:30:56.886936+0000 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:30:56.887 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.886+0000 7feeb2315640 1 Processor -- start 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.886+0000 7feeb2315640 1 -- start start 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.886+0000 7feeb2315640 10 monclient: init 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-c con 0x7feeac0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-b con 0x7feeac13f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-a con 0x7feeac1430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac142520 con 0x7feeac1430c0 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0b28a0 con 0x7feeac0a98f0 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0b2a30 con 0x7feeac13f710 2024-03-25T17:30:56.889 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:56.890 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeb2315640 10 monclient(hunting): authenticate will time out at 5415.169232s 2024-03-25T17:30:56.890 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeabfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7feeac0a98f0 0x7feeac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:38108/0 (socket says 172.21.15.96:38108) 2024-03-25T17:30:56.890 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.887+0000 7feeabfff640 1 -- 172.21.15.96:0/944308778 learned_addr learned my addr 172.21.15.96:0/944308778 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:56.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877097041 0 0) 0x7feeac142520 con 0x7feeac1430c0 2024-03-25T17:30:56.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:56.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:56.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:56.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T17:30:56.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.916+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003520 con 0x7feeac1430c0 2024-03-25T17:30:56.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526919533 0 0) 0x7fee98003520 con 0x7feeac1430c0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 >> v1:172.21.15.96:6790/0 conn(0x7feeac13f710 legacy=0x7feeac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 >> v1:172.21.15.203:6789/0 conn(0x7feeac0a98f0 legacy=0x7feeac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 monclient: found mon.noname-a 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0b44c0 con 0x7feeac1430c0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: _finish_auth 0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7feea4001a10 con 0x7feeac1430c0 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:56.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeab7fe640 10 monclient: dump: 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeb2315640 5 monclient: authenticate success, global_id 4305 2024-03-25T17:30:56.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.927+0000 7feeb2315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea4001b70 con 0x7feeac1430c0 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 1 -- 172.21.15.96:0/944308778 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7feea4001f80 con 0x7feeac1430c0 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeab7fe640 10 monclient: dump: 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.940 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeb2315640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:56.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.939+0000 7feeb2315640 4 set_mon_vals no callback set 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 10 monclient: shutdown 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 1 -- 172.21.15.96:0/944308778 >> v1:172.21.15.96:6789/0 conn(0x7feeac1430c0 legacy=0x7feeac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 1 -- 172.21.15.96:0/944308778 shutdown_connections 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 1 -- 172.21.15.96:0/944308778 >> 172.21.15.96:0/944308778 conn(0x7feeac0b7ea0 msgr2=0x7feeac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 1 -- 172.21.15.96:0/944308778 shutdown_connections 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.940+0000 7feeb2315640 1 -- 172.21.15.96:0/944308778 wait complete. 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 10 monclient: build_initial_monmap 2024-03-25T17:30:56.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 10 monclient: monmap: 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 librados: starting msgr at 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 librados: starting objecter 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 Processor -- start 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 -- start start 2024-03-25T17:30:56.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 librados: setting wanted keys 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 1 librados: calling monclient init 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.941+0000 7feeb2315640 10 monclient: init 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-c con 0x7feeac0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-a con 0x7feeac13f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient(hunting): picked mon.noname-b con 0x7feeac1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac139a10 con 0x7feeac13f710 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac1d7880 con 0x7feeac1430c0 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac137890 con 0x7feeac0a98f0 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:56.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.942+0000 7feeb2315640 10 monclient(hunting): authenticate will time out at 5415.224670s 2024-03-25T17:30:56.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.953+0000 7feea3fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7feeac13f710 0x7feeac138370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44996/0 (socket says 172.21.15.96:44996) 2024-03-25T17:30:56.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.953+0000 7feea3fff640 1 -- 172.21.15.96:0/988658693 learned_addr learned my addr 172.21.15.96:0/988658693 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:30:56.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.957+0000 7feea97fa640 10 client.?.objecter ms_handle_connect 0x7feeac13f710 2024-03-25T17:30:56.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.957+0000 7feea97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:56.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254517648 0 0) 0x7feeac139a10 con 0x7feeac13f710 2024-03-25T17:30:56.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:30:56.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:56.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:56.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 10 monclient(hunting): my global_id is 4308 2024-03-25T17:30:56.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.964+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee8c004eb0 con 0x7feeac13f710 2024-03-25T17:30:56.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.975+0000 7feea97fa640 10 client.?.objecter ms_handle_connect 0x7feeac1430c0 2024-03-25T17:30:56.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.975+0000 7feea97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:56.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.976+0000 7feea97fa640 10 client.?.objecter ms_handle_connect 0x7feeac0a98f0 2024-03-25T17:30:56.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.976+0000 7feea97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:56.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.980+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703167073 0 0) 0x7fee8c004eb0 con 0x7feeac13f710 2024-03-25T17:30:56.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.980+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeac139a10 con 0x7feeac13f710 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.981+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee940019d0 con 0x7feeac13f710 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.981+0000 7feea97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.981+0000 7feea97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.981+0000 7feea97fa640 10 monclient(hunting): dump: 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:56.982 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:56.983 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:56.983 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.981+0000 7feea97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4035460730 0 0) 0x7feeac139a10 con 0x7feeac13f710 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 >> v1:172.21.15.96:6790/0 conn(0x7feeac0a98f0 legacy=0x7feeac1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 >> v1:172.21.15.203:6789/0 conn(0x7feeac1430c0 legacy=0x7feeac1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 1 monclient: found mon.a 2024-03-25T17:30:56.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac1d84b0 con 0x7feeac13f710 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 10 monclient: _finish_auth 0 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feea97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 5 monclient: authenticate success, global_id 4308 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 10 monclient: _renew_subs 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac1d51f0 con 0x7feeac13f710 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 10 monclient: _renew_subs 2024-03-25T17:30:56.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:56.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac138c60 con 0x7feeac13f710 2024-03-25T17:30:56.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.995+0000 7feeb2315640 1 librados: init done 2024-03-25T17:30:56.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.996+0000 7feeb2315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:56.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.996+0000 7feeb2315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:56.999 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.996+0000 7feeb2315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:56.999 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:56.996+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac0627b0 con 0x7feeac13f710 2024-03-25T17:30:57.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee94001c50 con 0x7feeac13f710 2024-03-25T17:30:57.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee94003a60 con 0x7feeac13f710 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feea97fa640 10 monclient: dump: 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:30:57.302 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:30:57.303 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:30:57.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.300+0000 7feeaaffd640 4 set_mon_vals no callback set 2024-03-25T17:30:57.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.301+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fee94003c80 con 0x7feeac13f710 2024-03-25T17:30:57.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.302+0000 7feeabfff640 10 monclient: get_auth_request con 0x7fee8c06dd90 auth_method 0 2024-03-25T17:30:57.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.302+0000 7feea97fa640 10 client.4308.objecter ms_handle_connect 0x7fee8c06dd90 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fee94054830 con 0x7feeac13f710 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 10 client.4308.objecter ms_dispatch 0x7feeac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-25T17:30:57.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee94050340 con 0x7feeac13f710 2024-03-25T17:30:57.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:57.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.306+0000 7feea97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:57.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.551+0000 7feeb2315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:57.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.551+0000 7feeb2315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:57.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.551+0000 7feeb2315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:30:57.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.551+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeac13df90 con 0x7feeac13f710 2024-03-25T17:30:57.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.580+0000 7feea97fa640 1 -- 172.21.15.96:0/988658693 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fee94052270 con 0x7feeac13f710 2024-03-25T17:30:57.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.580+0000 7feea97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:57.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.580+0000 7feea97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:57.582 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:30:57.582 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:57.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 10 librados: watch_flush enter 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 10 librados: watch_flush exit 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 10 client.4308.objecter successfully canceled tick 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 >> v1:172.21.15.203:6800/3611964061 conn(0x7fee8c06dd90 legacy=0x7fee8c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 10 monclient: shutdown 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 >> v1:172.21.15.96:6789/0 conn(0x7feeac13f710 legacy=0x7feeac138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 shutdown_connections 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 >> 172.21.15.96:0/988658693 conn(0x7feeac0b7ea0 msgr2=0x7feeac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 shutdown_connections 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 -- 172.21.15.96:0/988658693 wait complete. 2024-03-25T17:30:57.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:30:57.583+0000 7feeb2315640 1 librados: shutdown 2024-03-25T17:30:57.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:00.039 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:24.265 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.044 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:24.270 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.119 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:24.345 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.154 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:24.370 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.159 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:24.374 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.173 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:24.400 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.178 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:24.405 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.190 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:24.405 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:00.211 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:24.438 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.403 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:04.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.523+0000 7f778c43e640 10 monclient: get_monmap_and_config 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.523+0000 7f778c43e640 10 monclient: build_initial_monmap 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.523+0000 7f778c43e640 10 monclient: monmap: 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:04.524511+0000 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:04.524511+0000 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:04.524 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.523+0000 7f778c43e640 1 Processor -- start 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 1 -- start start 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient: init 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-c con 0x7f77840b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-a con 0x7f77840b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-b con 0x7f778414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840af940 con 0x7f77840b1730 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840b0a70 con 0x7f77840b4d60 2024-03-25T17:31:04.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840b0d50 con 0x7f778414be90 2024-03-25T17:31:04.527 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:04.527 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.524+0000 7f778c43e640 10 monclient(hunting): authenticate will time out at 5422.806841s 2024-03-25T17:31:04.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.537+0000 7f778a1da640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f77840b4d60 0x7f778413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:47054/0 (socket says 172.21.15.96:47054) 2024-03-25T17:31:04.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.537+0000 7f778a1da640 1 -- 172.21.15.96:0/2820785931 learned_addr learned my addr 172.21.15.96:0/2820785931 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:04.545 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088215548 0 0) 0x7f77840b0d50 con 0x7f778414be90 2024-03-25T17:31:04.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 10 monclient(hunting): my global_id is 4304 2024-03-25T17:31:04.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.545+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7770003520 con 0x7f778414be90 2024-03-25T17:31:04.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008313076 0 0) 0x7f77840b0a70 con 0x7f77840b4d60 2024-03-25T17:31:04.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 10 monclient(hunting): my global_id is 4297 2024-03-25T17:31:04.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.550+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77840b0d50 con 0x7f77840b4d60 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721177361 0 0) 0x7f77840af940 con 0x7f77840b1730 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 10 monclient(hunting): my global_id is 4311 2024-03-25T17:31:04.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.554+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77840b0a70 con 0x7f77840b1730 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2324548158 0 0) 0x7f7770003520 con 0x7f778414be90 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 >> v1:172.21.15.203:6789/0 conn(0x7f77840b4d60 legacy=0x7f778413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 >> v1:172.21.15.96:6789/0 conn(0x7f77840b1730 legacy=0x7f77840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: _finish_hunting 0 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 monclient: found mon.noname-b 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:04.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840b1030 con 0x7f778414be90 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: _finish_auth 0 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: _check_auth_tickets 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f777c001a10 con 0x7f778414be90 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f778c43e640 5 monclient: authenticate success, global_id 4304 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f778c43e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f77891d8640 10 monclient: dump: 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.562 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.560+0000 7f778c43e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f777c001b70 con 0x7f778414be90 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 1 -- 172.21.15.96:0/2820785931 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f777c001f80 con 0x7f778414be90 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f77891d8640 10 monclient: dump: 2024-03-25T17:31:04.582 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f778c43e640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:04.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.581+0000 7f778c43e640 4 set_mon_vals no callback set 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 10 monclient: shutdown 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 1 -- 172.21.15.96:0/2820785931 >> v1:172.21.15.96:6790/0 conn(0x7f778414be90 legacy=0x7f778414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 1 -- 172.21.15.96:0/2820785931 shutdown_connections 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 1 -- 172.21.15.96:0/2820785931 >> 172.21.15.96:0/2820785931 conn(0x7f77840617a0 msgr2=0x7f7784061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.582+0000 7f778c43e640 1 -- 172.21.15.96:0/2820785931 shutdown_connections 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.583+0000 7f778c43e640 1 -- 172.21.15.96:0/2820785931 wait complete. 2024-03-25T17:31:04.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.583+0000 7f778c43e640 10 monclient: build_initial_monmap 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.583+0000 7f778c43e640 10 monclient: monmap: 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:04.585 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 librados: starting msgr at 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 librados: starting objecter 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 Processor -- start 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 -- start start 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 librados: setting wanted keys 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 1 librados: calling monclient init 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.584+0000 7f778c43e640 10 monclient: init 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-b con 0x7f77840b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-c con 0x7f77840b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient(hunting): picked mon.noname-a con 0x7f778414be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7784141a50 con 0x7f778414be90 2024-03-25T17:31:04.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7784135810 con 0x7f77840b1730 2024-03-25T17:31:04.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7784141cc0 con 0x7f77840b4d60 2024-03-25T17:31:04.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:04.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.585+0000 7f778c43e640 10 monclient(hunting): authenticate will time out at 5422.867439s 2024-03-25T17:31:04.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.597+0000 7f77899d9640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f77840b4d60 0x7f77841349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:44414/0 (socket says 172.21.15.96:44414) 2024-03-25T17:31:04.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.597+0000 7f77899d9640 1 -- 172.21.15.96:0/363371209 learned_addr learned my addr 172.21.15.96:0/363371209 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:04.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.599+0000 7f777affd640 10 client.?.objecter ms_handle_connect 0x7f77840b4d60 2024-03-25T17:31:04.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.599+0000 7f777affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:04.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.606+0000 7f777affd640 10 client.?.objecter ms_handle_connect 0x7f778414be90 2024-03-25T17:31:04.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.606+0000 7f777affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912876115 0 0) 0x7f7784141cc0 con 0x7f77840b4d60 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 monclient(hunting): my global_id is 4307 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768004eb0 con 0x7f77840b4d60 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 client.?.objecter ms_handle_connect 0x7f77840b1730 2024-03-25T17:31:04.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.608+0000 7f777affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576317660 0 0) 0x7f7768004eb0 con 0x7f77840b4d60 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7784141cc0 con 0x7f77840b4d60 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7774001a10 con 0x7f77840b4d60 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 10 monclient(hunting): dump: 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:04.619 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:04.620 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:04.620 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.618+0000 7f777affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:04.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115419660 0 0) 0x7f7784141a50 con 0x7f778414be90 2024-03-25T17:31:04.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): my global_id is 4314 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768004eb0 con 0x7f778414be90 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997409087 0 0) 0x7f7784135810 con 0x7f77840b1730 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 10 monclient(hunting): my global_id is 4300 2024-03-25T17:31:04.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.620+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7784141a50 con 0x7f77840b1730 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963528465 0 0) 0x7f7768004eb0 con 0x7f778414be90 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7784135810 con 0x7f778414be90 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f777c001e80 con 0x7f778414be90 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient(hunting): dump: 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.631 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402428778 0 0) 0x7f7784141cc0 con 0x7f77840b4d60 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 >> v1:172.21.15.203:6789/0 conn(0x7f77840b1730 legacy=0x7f77841dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 >> v1:172.21.15.96:6789/0 conn(0x7f778414be90 legacy=0x7f7784135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient: _finish_hunting 0 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 monclient: found mon.c 2024-03-25T17:31:04.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7784140810 con 0x7f77840b4d60 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient: _finish_auth 0 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 10 monclient: _check_auth_tickets 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f777affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 5 monclient: authenticate success, global_id 4307 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 10 monclient: _renew_subs 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77841377a0 con 0x7f77840b4d60 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 10 monclient: _renew_subs 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7784140560 con 0x7f77840b4d60 2024-03-25T17:31:04.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.629+0000 7f778c43e640 1 librados: init done 2024-03-25T17:31:04.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.631+0000 7f778c43e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:04.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.631+0000 7f778c43e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:04.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.631+0000 7f778c43e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:04.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.631+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77840627b0 con 0x7f77840b4d60 2024-03-25T17:31:04.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.841+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7774001cb0 con 0x7f77840b4d60 2024-03-25T17:31:04.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.841+0000 7f777affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:04.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.841+0000 7f77889d7640 4 set_mon_vals no callback set 2024-03-25T17:31:04.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.916+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7774003a60 con 0x7f77840b4d60 2024-03-25T17:31:04.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.916+0000 7f777affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.916+0000 7f777affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.916+0000 7f777affd640 10 monclient: dump: 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:04.917 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:04.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.916+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7774003c40 con 0x7f77840b4d60 2024-03-25T17:31:04.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.917+0000 7f778a1da640 10 monclient: get_auth_request con 0x7f776806e020 auth_method 0 2024-03-25T17:31:04.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.917+0000 7f777affd640 10 client.4307.objecter ms_handle_connect 0x7f776806e020 2024-03-25T17:31:04.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f7774054930 con 0x7f77840b4d60 2024-03-25T17:31:04.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 10 client.4307.objecter ms_dispatch 0x7f7784000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:04.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:04.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:04.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T17:31:04.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.924+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7774050270 con 0x7f77840b4d60 2024-03-25T17:31:04.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.925+0000 7f777affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:04.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:04.925+0000 7f777affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:05.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.216+0000 7f778c43e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:05.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.216+0000 7f778c43e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:05.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.216+0000 7f778c43e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:05.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.216+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77841409a0 con 0x7f77840b4d60 2024-03-25T17:31:05.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.238+0000 7f777affd640 1 -- 172.21.15.96:0/363371209 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f7774052370 con 0x7f77840b4d60 2024-03-25T17:31:05.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.238+0000 7f777affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:05.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.238+0000 7f777affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:05.239 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:05.239 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:05.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 10 librados: watch_flush enter 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 10 librados: watch_flush exit 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 10 client.4307.objecter successfully canceled tick 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 >> v1:172.21.15.203:6800/3611964061 conn(0x7f776806e020 legacy=0x7f77680704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 10 monclient: shutdown 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 >> v1:172.21.15.96:6790/0 conn(0x7f77840b4d60 legacy=0x7f77841349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 shutdown_connections 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.240+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 >> 172.21.15.96:0/363371209 conn(0x7f77840617a0 msgr2=0x7f778413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.241+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 shutdown_connections 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.241+0000 7f778c43e640 1 -- 172.21.15.96:0/363371209 wait complete. 2024-03-25T17:31:05.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:05.241+0000 7f778c43e640 1 librados: shutdown 2024-03-25T17:31:05.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:12.061 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:12.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.175+0000 7fc78f38f640 10 monclient: get_monmap_and_config 2024-03-25T17:31:12.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.175+0000 7fc78f38f640 10 monclient: build_initial_monmap 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.175+0000 7fc78f38f640 10 monclient: monmap: 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:12.176509+0000 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:12.176509+0000 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:12.176 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.176+0000 7fc78f38f640 1 Processor -- start 2024-03-25T17:31:12.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.176+0000 7fc78f38f640 1 -- start start 2024-03-25T17:31:12.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.176+0000 7fc78f38f640 10 monclient: init 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7880a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-c con 0x7fc78813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7881430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc788142520 con 0x7fc7881430c0 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7880b28a0 con 0x7fc78813f710 2024-03-25T17:31:12.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7880b2a30 con 0x7fc7880a98f0 2024-03-25T17:31:12.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:12.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.177+0000 7fc78f38f640 10 monclient(hunting): authenticate will time out at 5430.459559s 2024-03-25T17:31:12.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.181+0000 7fc78c92a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc78813f710 0x7fc7881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:51656/0 (socket says 172.21.15.96:51656) 2024-03-25T17:31:12.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.181+0000 7fc78c92a640 1 -- 172.21.15.96:0/2847746998 learned_addr learned my addr 172.21.15.96:0/2847746998 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:12.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710749225 0 0) 0x7fc7880b28a0 con 0x7fc78813f710 2024-03-25T17:31:12.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 10 monclient(hunting): my global_id is 4303 2024-03-25T17:31:12.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.193+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774003520 con 0x7fc78813f710 2024-03-25T17:31:12.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792255618 0 0) 0x7fc788142520 con 0x7fc7881430c0 2024-03-25T17:31:12.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 10 monclient(hunting): my global_id is 4317 2024-03-25T17:31:12.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.201+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774003840 con 0x7fc7881430c0 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773968553 0 0) 0x7fc7880b2a30 con 0x7fc7880a98f0 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 10 monclient(hunting): my global_id is 4310 2024-03-25T17:31:12.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.205+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc788142520 con 0x7fc7880a98f0 2024-03-25T17:31:12.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107358726 0 0) 0x7fc788142520 con 0x7fc7880a98f0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 >> v1:172.21.15.203:6789/0 conn(0x7fc78813f710 legacy=0x7fc7881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 >> v1:172.21.15.96:6789/0 conn(0x7fc7881430c0 legacy=0x7fc78814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 monclient: found mon.noname-b 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7880b44c0 con 0x7fc7880a98f0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: _finish_auth 0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc784001630 con 0x7fc7880a98f0 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:12.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc77ffff640 10 monclient: dump: 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc78f38f640 5 monclient: authenticate success, global_id 4310 2024-03-25T17:31:12.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.211+0000 7fc78f38f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc784001790 con 0x7fc7880a98f0 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 1 -- 172.21.15.96:0/2847746998 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc784001b60 con 0x7fc7880a98f0 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc78f38f640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc78f38f640 4 set_mon_vals no callback set 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.231+0000 7fc77ffff640 10 monclient: dump: 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.232 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.233 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.233 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.233 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.233 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 10 monclient: shutdown 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 1 -- 172.21.15.96:0/2847746998 >> v1:172.21.15.96:6790/0 conn(0x7fc7880a98f0 legacy=0x7fc7880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 1 -- 172.21.15.96:0/2847746998 shutdown_connections 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 1 -- 172.21.15.96:0/2847746998 >> 172.21.15.96:0/2847746998 conn(0x7fc7880b7ea0 msgr2=0x7fc7880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 1 -- 172.21.15.96:0/2847746998 shutdown_connections 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.232+0000 7fc78f38f640 1 -- 172.21.15.96:0/2847746998 wait complete. 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 10 monclient: build_initial_monmap 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 10 monclient: monmap: 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:12.235 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 librados: starting msgr at 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 librados: starting objecter 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 Processor -- start 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 -- start start 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 librados: setting wanted keys 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 1 librados: calling monclient init 2024-03-25T17:31:12.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.233+0000 7fc78f38f640 10 monclient: init 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7880a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-b con 0x7fc78813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7881430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7881398a0 con 0x7fc7880a98f0 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7881d4bc0 con 0x7fc78813f710 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7880b16f0 con 0x7fc7881430c0 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:12.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.234+0000 7fc78f38f640 10 monclient(hunting): authenticate will time out at 5430.516330s 2024-03-25T17:31:12.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.245+0000 7fc78c92a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc78813f710 0x7fc7880afd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:51728/0 (socket says 172.21.15.96:51728) 2024-03-25T17:31:12.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.245+0000 7fc78c92a640 1 -- 172.21.15.96:0/4051152275 learned_addr learned my addr 172.21.15.96:0/4051152275 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:12.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.248+0000 7fc77dffb640 10 client.?.objecter ms_handle_connect 0x7fc78813f710 2024-03-25T17:31:12.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.248+0000 7fc77dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:12.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.248+0000 7fc77dffb640 10 client.?.objecter ms_handle_connect 0x7fc7881430c0 2024-03-25T17:31:12.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.248+0000 7fc77dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:12.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.255+0000 7fc77dffb640 10 client.?.objecter ms_handle_connect 0x7fc7880a98f0 2024-03-25T17:31:12.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.255+0000 7fc77dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020209726 0 0) 0x7fc7881d4bc0 con 0x7fc78813f710 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 10 monclient(hunting): my global_id is 4306 2024-03-25T17:31:12.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.259+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76c004eb0 con 0x7fc78813f710 2024-03-25T17:31:12.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225074693 0 0) 0x7fc7880b16f0 con 0x7fc7881430c0 2024-03-25T17:31:12.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 10 monclient(hunting): my global_id is 4313 2024-03-25T17:31:12.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.272+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7881d4bc0 con 0x7fc7881430c0 2024-03-25T17:31:12.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697723203 0 0) 0x7fc7881398a0 con 0x7fc7880a98f0 2024-03-25T17:31:12.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:12.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:12.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:12.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T17:31:12.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.274+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7880b16f0 con 0x7fc7880a98f0 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054001632 0 0) 0x7fc76c004eb0 con 0x7fc78813f710 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7881398a0 con 0x7fc78813f710 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc778001990 con 0x7fc78813f710 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 10 monclient(hunting): dump: 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.277 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.276+0000 7fc77dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695991586 0 0) 0x7fc7881d4bc0 con 0x7fc7881430c0 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc76c004eb0 con 0x7fc7881430c0 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc780001b90 con 0x7fc7881430c0 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.281+0000 7fc77dffb640 10 monclient(hunting): dump: 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.282 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923860488 0 0) 0x7fc7880b16f0 con 0x7fc7880a98f0 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7881d4bc0 con 0x7fc7880a98f0 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc784001aa0 con 0x7fc7880a98f0 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.283+0000 7fc77dffb640 10 monclient(hunting): dump: 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.284 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 895523250 0 0) 0x7fc7881398a0 con 0x7fc78813f710 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 >> v1:172.21.15.96:6790/0 conn(0x7fc7881430c0 legacy=0x7fc7880b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 >> v1:172.21.15.96:6789/0 conn(0x7fc7880a98f0 legacy=0x7fc7881d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 1 monclient: found mon.b 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7880b1880 con 0x7fc78813f710 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 10 monclient: _finish_auth 0 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:12.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 5 monclient: authenticate success, global_id 4306 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 10 monclient: _renew_subs 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7880b0c50 con 0x7fc78813f710 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 10 monclient: _renew_subs 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7881d7c70 con 0x7fc78813f710 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.287+0000 7fc78f38f640 1 librados: init done 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.288+0000 7fc78f38f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.288+0000 7fc78f38f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:12.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.288+0000 7fc78f38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:12.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.288+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7880b0ff0 con 0x7fc78813f710 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7780047e0 con 0x7fc78813f710 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc778004af0 con 0x7fc78813f710 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:12.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77dffb640 10 monclient: dump: 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:12.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.566+0000 7fc77f7fe640 4 set_mon_vals no callback set 2024-03-25T17:31:12.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.567+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fc77801aac0 con 0x7fc78813f710 2024-03-25T17:31:12.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.568+0000 7fc78d12b640 10 monclient: get_auth_request con 0x7fc76c06e0f0 auth_method 0 2024-03-25T17:31:12.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.568+0000 7fc77dffb640 10 client.4306.objecter ms_handle_connect 0x7fc76c06e0f0 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fc778054800 con 0x7fc78813f710 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 10 client.4306.objecter ms_dispatch 0x7fc788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc778050140 con 0x7fc78813f710 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:12.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.573+0000 7fc77dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:12.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.820+0000 7fc78f38f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:12.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.820+0000 7fc78f38f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:12.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.820+0000 7fc78f38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:12.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.820+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7881d7f70 con 0x7fc78813f710 2024-03-25T17:31:12.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.965+0000 7fc77dffb640 1 -- 172.21.15.96:0/4051152275 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fc778052240 con 0x7fc78813f710 2024-03-25T17:31:12.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.965+0000 7fc77dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:12.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.965+0000 7fc77dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:12.967 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:12.967 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:12.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 10 librados: watch_flush enter 2024-03-25T17:31:12.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 10 librados: watch_flush exit 2024-03-25T17:31:12.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T17:31:12.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 10 client.4306.objecter successfully canceled tick 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 >> v1:172.21.15.203:6800/3611964061 conn(0x7fc76c06e0f0 legacy=0x7fc76c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 10 monclient: shutdown 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 >> v1:172.21.15.203:6789/0 conn(0x7fc78813f710 legacy=0x7fc7880afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 shutdown_connections 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 >> 172.21.15.96:0/4051152275 conn(0x7fc7880b7ea0 msgr2=0x7fc788074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.968+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 shutdown_connections 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.969+0000 7fc78f38f640 1 -- 172.21.15.96:0/4051152275 wait complete. 2024-03-25T17:31:12.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:12.969+0000 7fc78f38f640 1 librados: shutdown 2024-03-25T17:31:12.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:19.788 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.900+0000 7fe664b25640 10 monclient: get_monmap_and_config 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.900+0000 7fe664b25640 10 monclient: build_initial_monmap 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.900+0000 7fe664b25640 10 monclient: monmap: 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:19.902011+0000 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:19.902011+0000 2024-03-25T17:31:19.901 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.901+0000 7fe664b25640 1 Processor -- start 2024-03-25T17:31:19.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.901+0000 7fe664b25640 1 -- start start 2024-03-25T17:31:19.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.901+0000 7fe664b25640 10 monclient: init 2024-03-25T17:31:19.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.902+0000 7fe664b25640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:19.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.902+0000 7fe664b25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.902+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-c con 0x7fe66006ea00 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.902+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-b con 0x7fe66013bae0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-a con 0x7fe66006ede0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66006fb80 con 0x7fe66006ede0 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe660070c50 con 0x7fe66006ea00 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe660070f30 con 0x7fe66013bae0 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:19.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.903+0000 7fe664b25640 10 monclient(hunting): authenticate will time out at 5438.185030s 2024-03-25T17:31:19.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.915+0000 7fe657fff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fe66013bae0 0x7fe66013e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:42766/0 (socket says 172.21.15.96:42766) 2024-03-25T17:31:19.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.915+0000 7fe657fff640 1 -- 172.21.15.96:0/3792590829 learned_addr learned my addr 172.21.15.96:0/3792590829 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:19.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381089879 0 0) 0x7fe660070c50 con 0x7fe66006ea00 2024-03-25T17:31:19.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:19.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:19.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:19.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T17:31:19.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.926+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64c003520 con 0x7fe66006ea00 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958892377 0 0) 0x7fe66006fb80 con 0x7fe66006ede0 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 10 monclient(hunting): my global_id is 4323 2024-03-25T17:31:19.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.930+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660070c50 con 0x7fe66006ede0 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890696978 0 0) 0x7fe660070f30 con 0x7fe66013bae0 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 10 monclient(hunting): my global_id is 4316 2024-03-25T17:31:19.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.933+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66006fb80 con 0x7fe66013bae0 2024-03-25T17:31:19.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437981243 0 0) 0x7fe66006fb80 con 0x7fe66013bae0 2024-03-25T17:31:19.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 >> v1:172.21.15.203:6789/0 conn(0x7fe66006ea00 legacy=0x7fe66013b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:19.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 >> v1:172.21.15.96:6789/0 conn(0x7fe66006ede0 legacy=0x7fe66013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:19.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: _finish_hunting 0 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 monclient: found mon.noname-b 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66006f570 con 0x7fe66013bae0 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: _finish_auth 0 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: _check_auth_tickets 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe664b25640 5 monclient: authenticate success, global_id 4316 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe664b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe6500014f0 con 0x7fe66013bae0 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe65dd9b640 10 monclient: dump: 2024-03-25T17:31:19.943 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:19.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.941+0000 7fe664b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe650001790 con 0x7fe66013bae0 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 1 -- 172.21.15.96:0/3792590829 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe650001b20 con 0x7fe66013bae0 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe65dd9b640 10 monclient: dump: 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe664b25640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.957+0000 7fe664b25640 4 set_mon_vals no callback set 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 10 monclient: shutdown 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 1 -- 172.21.15.96:0/3792590829 >> v1:172.21.15.96:6790/0 conn(0x7fe66013bae0 legacy=0x7fe66013e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 1 -- 172.21.15.96:0/3792590829 shutdown_connections 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 1 -- 172.21.15.96:0/3792590829 >> 172.21.15.96:0/3792590829 conn(0x7fe6600b0ad0 msgr2=0x7fe6600b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 1 -- 172.21.15.96:0/3792590829 shutdown_connections 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.958+0000 7fe664b25640 1 -- 172.21.15.96:0/3792590829 wait complete. 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.959+0000 7fe664b25640 10 monclient: build_initial_monmap 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.959+0000 7fe664b25640 10 monclient: monmap: 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:19.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.959+0000 7fe664b25640 1 librados: starting msgr at 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.959+0000 7fe664b25640 1 librados: starting objecter 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.959+0000 7fe664b25640 1 Processor -- start 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 -- start start 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 librados: setting wanted keys 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 librados: calling monclient init 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient: init 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-b con 0x7fe66006ea00 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-c con 0x7fe66006ede0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient(hunting): picked mon.noname-a con 0x7fe66013bae0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66013c110 con 0x7fe66013bae0 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe660142130 con 0x7fe66006ea00 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66013fe80 con 0x7fe66006ede0 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:19.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.960+0000 7fe664b25640 10 monclient(hunting): authenticate will time out at 5438.242747s 2024-03-25T17:31:19.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.971+0000 7fe65e59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe66006ea00 0x7fe6601403a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:45272/0 (socket says 172.21.15.96:45272) 2024-03-25T17:31:19.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.971+0000 7fe65e59c640 1 -- 172.21.15.96:0/3505826869 learned_addr learned my addr 172.21.15.96:0/3505826869 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:19.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.971+0000 7fe656ffd640 10 client.?.objecter ms_handle_connect 0x7fe66006ea00 2024-03-25T17:31:19.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.971+0000 7fe656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:19.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.975+0000 7fe656ffd640 10 client.?.objecter ms_handle_connect 0x7fe66006ede0 2024-03-25T17:31:19.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.975+0000 7fe656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:19.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124079248 0 0) 0x7fe660142130 con 0x7fe66006ea00 2024-03-25T17:31:19.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:19.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:19.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:19.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 10 monclient(hunting): my global_id is 4312 2024-03-25T17:31:19.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.979+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640004eb0 con 0x7fe66006ea00 2024-03-25T17:31:19.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.980+0000 7fe656ffd640 10 client.?.objecter ms_handle_connect 0x7fe66013bae0 2024-03-25T17:31:19.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.980+0000 7fe656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001502891 0 0) 0x7fe66013fe80 con 0x7fe66006ede0 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T17:31:19.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:19.983+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660142130 con 0x7fe66006ede0 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708315518 0 0) 0x7fe640004eb0 con 0x7fe66006ea00 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe66013fe80 con 0x7fe66006ea00 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe6480019b0 con 0x7fe66006ea00 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 10 monclient(hunting): dump: 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:20.003 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:20.004 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:20.004 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:20.004 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:20.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.002+0000 7fe656ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:20.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.004+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122984790 0 0) 0x7fe660142130 con 0x7fe66006ede0 2024-03-25T17:31:20.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.005+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe640004eb0 con 0x7fe66006ede0 2024-03-25T17:31:20.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.005+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe650001b60 con 0x7fe66006ede0 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.005+0000 7fe656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.005+0000 7fe656ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.005+0000 7fe656ffd640 10 monclient(hunting): dump: 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:20.006 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997054381 0 0) 0x7fe66013c110 con 0x7fe66013bae0 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T17:31:20.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.006+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660142130 con 0x7fe66013bae0 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1451872454 0 0) 0x7fe66013fe80 con 0x7fe66006ea00 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 >> v1:172.21.15.96:6790/0 conn(0x7fe66006ede0 legacy=0x7fe660133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 >> v1:172.21.15.96:6789/0 conn(0x7fe66013bae0 legacy=0x7fe6601e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 1 monclient: found mon.b 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6601d4f40 con 0x7fe66006ea00 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 10 monclient: _finish_auth 0 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:31:20.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 5 monclient: authenticate success, global_id 4312 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 10 monclient: _renew_subs 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6601db630 con 0x7fe66006ea00 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 10 monclient: _renew_subs 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66013fbd0 con 0x7fe66006ea00 2024-03-25T17:31:20.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.019+0000 7fe664b25640 1 librados: init done 2024-03-25T17:31:20.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.020+0000 7fe664b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:20.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.020+0000 7fe664b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:20.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.020+0000 7fe664b25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:20.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.020+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe660074d20 con 0x7fe66006ea00 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe648003610 con 0x7fe66006ea00 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe648003a20 con 0x7fe66006ea00 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe656ffd640 10 monclient: dump: 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:20.300 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:20.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.299+0000 7fe65d59a640 4 set_mon_vals no callback set 2024-03-25T17:31:20.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.300+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fe648003c40 con 0x7fe66006ea00 2024-03-25T17:31:20.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.300+0000 7fe657fff640 10 monclient: get_auth_request con 0x7fe64006df50 auth_method 0 2024-03-25T17:31:20.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.301+0000 7fe656ffd640 10 client.4312.objecter ms_handle_connect 0x7fe64006df50 2024-03-25T17:31:20.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.305+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fe6480547a0 con 0x7fe66006ea00 2024-03-25T17:31:20.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.305+0000 7fe656ffd640 10 client.4312.objecter ms_dispatch 0x7fe660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.305+0000 7fe656ffd640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.305+0000 7fe656ffd640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.306+0000 7fe656ffd640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.306+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6480502b0 con 0x7fe66006ea00 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.306+0000 7fe656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:20.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.306+0000 7fe656ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:20.362 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:44.588 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.408 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:44.624 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.410 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:44.636 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.412 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:44.627 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.425 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:44.641 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.441 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:02:44.668 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.516 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:44.742 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.526 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:44.752 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.542+0000 7fe664b25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:20.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.542+0000 7fe664b25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:20.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.542+0000 7fe664b25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:20.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.542+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6600627b0 con 0x7fe66006ea00 2024-03-25T17:31:20.554 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:02:44.781 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:20.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.575+0000 7fe656ffd640 1 -- 172.21.15.96:0/3505826869 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fe6480521e0 con 0x7fe66006ea00 2024-03-25T17:31:20.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.575+0000 7fe656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:20.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.575+0000 7fe656ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:20.576 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:20.576 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:20.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 10 librados: watch_flush enter 2024-03-25T17:31:20.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 10 librados: watch_flush exit 2024-03-25T17:31:20.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 10 client.4312.objecter successfully canceled tick 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 >> v1:172.21.15.203:6800/3611964061 conn(0x7fe64006df50 legacy=0x7fe640070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 10 monclient: shutdown 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 >> v1:172.21.15.203:6789/0 conn(0x7fe66006ea00 legacy=0x7fe6601403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 shutdown_connections 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 >> 172.21.15.96:0/3505826869 conn(0x7fe6600b0ad0 msgr2=0x7fe660071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 shutdown_connections 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 -- 172.21.15.96:0/3505826869 wait complete. 2024-03-25T17:31:20.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:20.577+0000 7fe664b25640 1 librados: shutdown 2024-03-25T17:31:20.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:27.397 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:27.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.512+0000 7f10cb255640 10 monclient: get_monmap_and_config 2024-03-25T17:31:27.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.512+0000 7f10cb255640 10 monclient: build_initial_monmap 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.513+0000 7f10cb255640 10 monclient: monmap: 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:27.513277+0000 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:27.513277+0000 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:27.513 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.513+0000 7f10cb255640 1 Processor -- start 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 1 -- start start 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient: init 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-b con 0x7f10c40b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:27.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-a con 0x7f10c40b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-c con 0x7f10c414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40af940 con 0x7f10c40b1730 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40b0a70 con 0x7f10c414be90 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40b0d50 con 0x7f10c40b4d60 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:27.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.514+0000 7f10cb255640 10 monclient(hunting): authenticate will time out at 5445.795659s 2024-03-25T17:31:27.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.517+0000 7f10c8ff1640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f10c40b4d60 0x7f10c413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:60110/0 (socket says 172.21.15.96:60110) 2024-03-25T17:31:27.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.517+0000 7f10c8ff1640 1 -- 172.21.15.96:0/1592418660 learned_addr learned my addr 172.21.15.96:0/1592418660 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988343068 0 0) 0x7f10c40b0d50 con 0x7f10c40b4d60 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T17:31:27.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.541+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4003520 con 0x7f10c40b4d60 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723862039 0 0) 0x7f10c40b0a70 con 0x7f10c414be90 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 10 monclient(hunting): my global_id is 4315 2024-03-25T17:31:27.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.543+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c40b0d50 con 0x7f10c414be90 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521938403 0 0) 0x7f10b4003520 con 0x7f10c40b4d60 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 >> v1:172.21.15.203:6789/0 conn(0x7f10c414be90 legacy=0x7f10c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 >> v1:172.21.15.96:6789/0 conn(0x7f10c40b1730 legacy=0x7f10c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 monclient: found mon.noname-b 2024-03-25T17:31:27.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c40b1030 con 0x7f10c40b4d60 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f10b8001690 con 0x7f10c40b4d60 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10cb255640 5 monclient: authenticate success, global_id 4322 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10cb255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10c37fe640 10 monclient: dump: 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.555 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.554+0000 7f10cb255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:27.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b80017f0 con 0x7f10c40b4d60 2024-03-25T17:31:27.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:27.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 1 -- 172.21.15.96:0/1592418660 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f10b8001c00 con 0x7f10c40b4d60 2024-03-25T17:31:27.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:27.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10c37fe640 10 monclient: dump: 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:27.563 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.564 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10cb255640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:27.564 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.562+0000 7f10cb255640 4 set_mon_vals no callback set 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 10 monclient: shutdown 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 1 -- 172.21.15.96:0/1592418660 >> v1:172.21.15.96:6790/0 conn(0x7f10c40b4d60 legacy=0x7f10c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 1 -- 172.21.15.96:0/1592418660 shutdown_connections 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 1 -- 172.21.15.96:0/1592418660 >> 172.21.15.96:0/1592418660 conn(0x7f10c40617a0 msgr2=0x7f10c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 1 -- 172.21.15.96:0/1592418660 shutdown_connections 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.563+0000 7f10cb255640 1 -- 172.21.15.96:0/1592418660 wait complete. 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.564+0000 7f10cb255640 10 monclient: build_initial_monmap 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.564+0000 7f10cb255640 10 monclient: monmap: 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:27.565 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 librados: starting msgr at 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 librados: starting objecter 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 Processor -- start 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 -- start start 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 librados: setting wanted keys 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 1 librados: calling monclient init 2024-03-25T17:31:27.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.565+0000 7f10cb255640 10 monclient: init 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-a con 0x7f10c40b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-c con 0x7f10c40b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient(hunting): picked mon.noname-b con 0x7f10c414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c4141bf0 con 0x7f10c40b1730 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c4141ed0 con 0x7f10c414be90 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c41421b0 con 0x7f10c40b4d60 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:27.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.566+0000 7f10cb255640 10 monclient(hunting): authenticate will time out at 5445.847587s 2024-03-25T17:31:27.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.573+0000 7f10c97f2640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f10c414be90 0x7f10c41341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:49178/0 (socket says 172.21.15.96:49178) 2024-03-25T17:31:27.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.573+0000 7f10c97f2640 1 -- 172.21.15.96:0/2464178272 learned_addr learned my addr 172.21.15.96:0/2464178272 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:27.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.578+0000 7f10c17fa640 10 client.?.objecter ms_handle_connect 0x7f10c414be90 2024-03-25T17:31:27.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.578+0000 7f10c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:27.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.583+0000 7f10c17fa640 10 client.?.objecter ms_handle_connect 0x7f10c40b4d60 2024-03-25T17:31:27.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.583+0000 7f10c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256195090 0 0) 0x7f10c4141ed0 con 0x7f10c414be90 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 10 monclient(hunting): my global_id is 4318 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.591+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a8004eb0 con 0x7f10c414be90 2024-03-25T17:31:27.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.592+0000 7f10c17fa640 10 client.?.objecter ms_handle_connect 0x7f10c40b1730 2024-03-25T17:31:27.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.592+0000 7f10c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676192864 0 0) 0x7f10c41421b0 con 0x7f10c40b4d60 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 10 monclient(hunting): my global_id is 4325 2024-03-25T17:31:27.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.601+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c4141ed0 con 0x7f10c40b4d60 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239491575 0 0) 0x7f10c4141bf0 con 0x7f10c40b1730 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 10 monclient(hunting): my global_id is 4332 2024-03-25T17:31:27.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.604+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c41421b0 con 0x7f10c40b1730 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210934327 0 0) 0x7f10c4141ed0 con 0x7f10c40b4d60 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c4141bf0 con 0x7f10c40b4d60 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f10b00018b0 con 0x7f10c40b4d60 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 10 monclient(hunting): dump: 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:27.609 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:27.610 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.609+0000 7f10c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047271869 0 0) 0x7f10a8004eb0 con 0x7f10c414be90 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c4141ed0 con 0x7f10c414be90 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f10bc001c70 con 0x7f10c414be90 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.613+0000 7f10c17fa640 10 monclient(hunting): dump: 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:27.613 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.614 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:27.614 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:27.614 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:27.614 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4013310235 0 0) 0x7f10c4141bf0 con 0x7f10c40b4d60 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 >> v1:172.21.15.203:6789/0 conn(0x7f10c414be90 legacy=0x7f10c41341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 >> v1:172.21.15.96:6789/0 conn(0x7f10c40b1730 legacy=0x7f10c41425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 1 monclient: found mon.c 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:27.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4140f00 con 0x7f10c40b4d60 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 10 monclient: _finish_auth 0 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 5 monclient: authenticate success, global_id 4325 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 10 monclient: _renew_subs 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c41406f0 con 0x7f10c40b4d60 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 10 monclient: _renew_subs 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c4140380 con 0x7f10c40b4d60 2024-03-25T17:31:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.620+0000 7f10cb255640 1 librados: init done 2024-03-25T17:31:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.621+0000 7f10cb255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.621+0000 7f10cb255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.621+0000 7f10cb255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.621+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c41401f0 con 0x7f10c40b4d60 2024-03-25T17:31:27.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b0003610 con 0x7f10c40b4d60 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f10b0003a60 con 0x7f10c40b4d60 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 10 monclient: dump: 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:27.922 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c27fc640 4 set_mon_vals no callback set 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.922+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f10b0003c80 con 0x7f10c40b4d60 2024-03-25T17:31:27.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.923+0000 7f10c8ff1640 10 monclient: get_auth_request con 0x7f10a806df70 auth_method 0 2024-03-25T17:31:27.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.923+0000 7f10c17fa640 10 client.4325.objecter ms_handle_connect 0x7f10a806df70 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f10b00535b0 con 0x7f10c40b4d60 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 10 client.4325.objecter ms_dispatch 0x7f10c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10b004f0c0 con 0x7f10c40b4d60 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:27.929+0000 7f10c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:28.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.166+0000 7f10cb255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:28.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.166+0000 7f10cb255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:28.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.166+0000 7f10cb255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:28.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.166+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10c40627b0 con 0x7f10c40b4d60 2024-03-25T17:31:28.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.195+0000 7f10c17fa640 1 -- 172.21.15.96:0/2464178272 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f10b0050ff0 con 0x7f10c40b4d60 2024-03-25T17:31:28.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.196+0000 7f10c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:28.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.196+0000 7f10c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:28.196 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:28.196 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:28.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 10 librados: watch_flush enter 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 10 librados: watch_flush exit 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 10 client.4325.objecter successfully canceled tick 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 >> v1:172.21.15.203:6800/3611964061 conn(0x7f10a806df70 legacy=0x7f10a8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 10 monclient: shutdown 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 >> v1:172.21.15.96:6790/0 conn(0x7f10c40b4d60 legacy=0x7f10c4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 shutdown_connections 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 >> 172.21.15.96:0/2464178272 conn(0x7f10c40617a0 msgr2=0x7f10c413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 shutdown_connections 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 -- 172.21.15.96:0/2464178272 wait complete. 2024-03-25T17:31:28.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:28.198+0000 7f10cb255640 1 librados: shutdown 2024-03-25T17:31:28.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:35.016 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.115+0000 7fb8c36ba640 10 monclient: get_monmap_and_config 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.115+0000 7fb8c36ba640 10 monclient: build_initial_monmap 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.115+0000 7fb8c36ba640 10 monclient: monmap: 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:35.116820+0000 2024-03-25T17:31:35.116 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:35.116820+0000 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:35.117 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 1 Processor -- start 2024-03-25T17:31:35.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 1 -- start start 2024-03-25T17:31:35.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient: init 2024-03-25T17:31:35.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:35.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb8bc14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0af940 con 0x7fb8bc14be90 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0b0a70 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0b0d50 con 0x7fb8bc0b1730 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:35.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.116+0000 7fb8c36ba640 10 monclient(hunting): authenticate will time out at 5453.398936s 2024-03-25T17:31:35.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.129+0000 7fb8c1456640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb8bc0b4d60 0x7fb8bc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:47164/0 (socket says 172.21.15.96:47164) 2024-03-25T17:31:35.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.129+0000 7fb8c1456640 1 -- 172.21.15.96:0/3032264108 learned_addr learned my addr 172.21.15.96:0/3032264108 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249254157 0 0) 0x7fb8bc0b0d50 con 0x7fb8bc0b1730 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 10 monclient(hunting): my global_id is 4328 2024-03-25T17:31:35.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.145+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a8003520 con 0x7fb8bc0b1730 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1358079243 0 0) 0x7fb8a8003520 con 0x7fb8bc0b1730 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 >> v1:172.21.15.203:6789/0 conn(0x7fb8bc0b4d60 legacy=0x7fb8bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 >> v1:172.21.15.96:6789/0 conn(0x7fb8bc14be90 legacy=0x7fb8bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 monclient: found mon.noname-b 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0b1030 con 0x7fb8bc0b1730 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 10 monclient: _finish_auth 0 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb8ac001690 con 0x7fb8bc0b1730 2024-03-25T17:31:35.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.162+0000 7fb8b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.163+0000 7fb8b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.163+0000 7fb8b3fff640 10 monclient: dump: 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.163+0000 7fb8c36ba640 5 monclient: authenticate success, global_id 4328 2024-03-25T17:31:35.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.163+0000 7fb8c36ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8ac0017f0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 1 -- 172.21.15.96:0/3032264108 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb8ac001c00 con 0x7fb8bc0b1730 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8c36ba640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8c36ba640 4 set_mon_vals no callback set 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.190+0000 7fb8b3fff640 10 monclient: dump: 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.191 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:35.192 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.192 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:35.192 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:35.192 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:35.192 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 10 monclient: shutdown 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 1 -- 172.21.15.96:0/3032264108 >> v1:172.21.15.96:6790/0 conn(0x7fb8bc0b1730 legacy=0x7fb8bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 1 -- 172.21.15.96:0/3032264108 shutdown_connections 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 1 -- 172.21.15.96:0/3032264108 >> 172.21.15.96:0/3032264108 conn(0x7fb8bc0617a0 msgr2=0x7fb8bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 1 -- 172.21.15.96:0/3032264108 shutdown_connections 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.191+0000 7fb8c36ba640 1 -- 172.21.15.96:0/3032264108 wait complete. 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 10 monclient: build_initial_monmap 2024-03-25T17:31:35.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 10 monclient: monmap: 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 librados: starting msgr at 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 librados: starting objecter 2024-03-25T17:31:35.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 Processor -- start 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 -- start start 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 librados: setting wanted keys 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 1 librados: calling monclient init 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.192+0000 7fb8c36ba640 10 monclient: init 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb8bc14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc1d79a0 con 0x7fb8bc14be90 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc136ed0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc138280 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:35.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.193+0000 7fb8c36ba640 10 monclient(hunting): authenticate will time out at 5453.475324s 2024-03-25T17:31:35.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.212+0000 7fb8c0c55640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fb8bc0b4d60 0x7fb8bc1d7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:58714/0 (socket says 172.21.15.96:58714) 2024-03-25T17:31:35.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.212+0000 7fb8c0c55640 1 -- 172.21.15.96:0/2707831830 learned_addr learned my addr 172.21.15.96:0/2707831830 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:35.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.215+0000 7fb8b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8bc0b4d60 2024-03-25T17:31:35.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.215+0000 7fb8b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:35.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.229+0000 7fb8b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8bc0b1730 2024-03-25T17:31:35.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.229+0000 7fb8b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:35.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394118580 0 0) 0x7fb8bc138280 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:35.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:35.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:35.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 10 monclient(hunting): my global_id is 4331 2024-03-25T17:31:35.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.231+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0004eb0 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.233+0000 7fb8b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8bc14be90 2024-03-25T17:31:35.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.233+0000 7fb8b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121451318 0 0) 0x7fb8bc136ed0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T17:31:35.241 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.240+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc138280 con 0x7fb8bc0b1730 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976783457 0 0) 0x7fb8bc1d79a0 con 0x7fb8bc14be90 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 10 monclient(hunting): my global_id is 4338 2024-03-25T17:31:35.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.245+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc136ed0 con 0x7fb8bc14be90 2024-03-25T17:31:35.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187085014 0 0) 0x7fb8a0004eb0 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8bc1d79a0 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb8ac001c00 con 0x7fb8bc0b4d60 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): dump: 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809180794 0 0) 0x7fb8bc138280 con 0x7fb8bc0b1730 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a0004eb0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb8b80018a0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.250+0000 7fb8b1ffb640 10 monclient(hunting): dump: 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:35.253 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:35.254 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:35.254 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1282389902 0 0) 0x7fb8a0004eb0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 >> v1:172.21.15.96:6790/0 conn(0x7fb8bc0b4d60 legacy=0x7fb8bc1d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 >> v1:172.21.15.96:6789/0 conn(0x7fb8bc14be90 legacy=0x7fb8bc1e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 1 monclient: found mon.b 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc143900 con 0x7fb8bc0b1730 2024-03-25T17:31:35.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 5 monclient: authenticate success, global_id 4324 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 10 monclient: _renew_subs 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc136000 con 0x7fb8bc0b1730 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 10 monclient: _renew_subs 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc1d8720 con 0x7fb8bc0b1730 2024-03-25T17:31:35.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.268+0000 7fb8c36ba640 1 librados: init done 2024-03-25T17:31:35.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.269+0000 7fb8c36ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:35.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.269+0000 7fb8c36ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:35.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.269+0000 7fb8c36ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:35.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.269+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc1401f0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8001cb0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb8b8004980 con 0x7fb8bc0b1730 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b1ffb640 10 monclient: dump: 2024-03-25T17:31:35.576 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.575+0000 7fb8b37fe640 4 set_mon_vals no callback set 2024-03-25T17:31:35.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.576+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fb8b8004b60 con 0x7fb8bc0b1730 2024-03-25T17:31:35.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.576+0000 7fb8c0c55640 10 monclient: get_auth_request con 0x7fb8a006dfb0 auth_method 0 2024-03-25T17:31:35.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.577+0000 7fb8b1ffb640 10 client.4324.objecter ms_handle_connect 0x7fb8a006dfb0 2024-03-25T17:31:35.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.581+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fb8b80546d0 con 0x7fb8bc0b1730 2024-03-25T17:31:35.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.581+0000 7fb8b1ffb640 10 client.4324.objecter ms_dispatch 0x7fb8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:35.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.581+0000 7fb8b1ffb640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:35.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.581+0000 7fb8b1ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:35.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.581+0000 7fb8b1ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T17:31:35.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.582+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8b8050010 con 0x7fb8bc0b1730 2024-03-25T17:31:35.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.582+0000 7fb8b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:35.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.582+0000 7fb8b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:35.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.830+0000 7fb8c36ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:35.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.830+0000 7fb8c36ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:35.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.830+0000 7fb8c36ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:35.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.830+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8bc138080 con 0x7fb8bc0b1730 2024-03-25T17:31:35.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.855+0000 7fb8b1ffb640 1 -- 172.21.15.96:0/2707831830 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fb8b8052110 con 0x7fb8bc0b1730 2024-03-25T17:31:35.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.855+0000 7fb8b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:35.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.855+0000 7fb8b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:35.857 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:35.857 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 10 librados: watch_flush enter 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 10 librados: watch_flush exit 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 10 client.4324.objecter successfully canceled tick 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 >> v1:172.21.15.203:6800/3611964061 conn(0x7fb8a006dfb0 legacy=0x7fb8a0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 10 monclient: shutdown 2024-03-25T17:31:35.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 >> v1:172.21.15.203:6789/0 conn(0x7fb8bc0b1730 legacy=0x7fb8bc1367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 shutdown_connections 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 >> 172.21.15.96:0/2707831830 conn(0x7fb8bc0617a0 msgr2=0x7fb8bc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.858+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 shutdown_connections 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.859+0000 7fb8c36ba640 1 -- 172.21.15.96:0/2707831830 wait complete. 2024-03-25T17:31:35.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:35.859+0000 7fb8c36ba640 1 librados: shutdown 2024-03-25T17:31:35.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:40.234 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:04.461 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.286 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:04.512 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.326 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:04.553 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.336 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:04.551 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.348 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:04.563 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.352 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:04.578 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.355 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:04.582 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.374 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:04.589 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:40.393 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:04.620 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:42.679 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.795+0000 7f22066c8640 10 monclient: get_monmap_and_config 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.795+0000 7f22066c8640 10 monclient: build_initial_monmap 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.795+0000 7f22066c8640 10 monclient: monmap: 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:42.796 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:42.796719+0000 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:42.796719+0000 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:42.797 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:42.798 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.796+0000 7f22066c8640 1 Processor -- start 2024-03-25T17:31:42.798 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.796+0000 7f22066c8640 1 -- start start 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.796+0000 7f22066c8640 10 monclient: init 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-b con 0x7f22000a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-c con 0x7f220013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-a con 0x7f22001430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200142520 con 0x7f22001430c0 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b28a0 con 0x7f220013f710 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b2a30 con 0x7f22000a98f0 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:42.799 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.797+0000 7f22066c8640 10 monclient(hunting): authenticate will time out at 5461.079106s 2024-03-25T17:31:42.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.805+0000 7f21fffff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f22000a98f0 0x7f22000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:35000/0 (socket says 172.21.15.96:35000) 2024-03-25T17:31:42.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.805+0000 7f21fffff640 1 -- 172.21.15.96:0/2506467971 learned_addr learned my addr 172.21.15.96:0/2506467971 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:42.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861031410 0 0) 0x7f22000b2a30 con 0x7f22000a98f0 2024-03-25T17:31:42.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:42.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:42.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:42.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 10 monclient(hunting): my global_id is 4334 2024-03-25T17:31:42.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.812+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e8003520 con 0x7f22000a98f0 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055497176 0 0) 0x7f2200142520 con 0x7f22001430c0 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T17:31:42.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.846+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22000b2a30 con 0x7f22001430c0 2024-03-25T17:31:42.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 273798555 0 0) 0x7f22000b2a30 con 0x7f22001430c0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 >> v1:172.21.15.96:6790/0 conn(0x7f22000a98f0 legacy=0x7f22000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 >> v1:172.21.15.203:6789/0 conn(0x7f220013f710 legacy=0x7f22001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: _finish_hunting 0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 monclient: found mon.noname-a 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000b44c0 con 0x7f22001430c0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: _finish_auth 0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: _check_auth_tickets 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f80019f0 con 0x7f22001430c0 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:42.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f21feffd640 10 monclient: dump: 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f22066c8640 5 monclient: authenticate success, global_id 4341 2024-03-25T17:31:42.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.927+0000 7f22066c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f8001b50 con 0x7f22001430c0 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f22066c8640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 1 -- 172.21.15.96:0/2506467971 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f8001f20 con 0x7f22001430c0 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f22066c8640 4 set_mon_vals no callback set 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:42.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.941+0000 7f21feffd640 10 monclient: dump: 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:42.943 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 10 monclient: shutdown 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 1 -- 172.21.15.96:0/2506467971 >> v1:172.21.15.96:6789/0 conn(0x7f22001430c0 legacy=0x7f220014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 1 -- 172.21.15.96:0/2506467971 shutdown_connections 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 1 -- 172.21.15.96:0/2506467971 >> 172.21.15.96:0/2506467971 conn(0x7f22000b7ea0 msgr2=0x7f22000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:42.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 1 -- 172.21.15.96:0/2506467971 shutdown_connections 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.942+0000 7f22066c8640 1 -- 172.21.15.96:0/2506467971 wait complete. 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 10 monclient: build_initial_monmap 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 10 monclient: monmap: 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:42.946 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 librados: starting msgr at 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 librados: starting objecter 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 Processor -- start 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 -- start start 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 librados: setting wanted keys 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 1 librados: calling monclient init 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.943+0000 7f22066c8640 10 monclient: init 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-a con 0x7f22000a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-c con 0x7f220013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient(hunting): picked mon.noname-b con 0x7f22001430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b7aa0 con 0x7f22000a98f0 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220013b3f0 con 0x7f22001430c0 2024-03-25T17:31:42.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220013b6d0 con 0x7f220013f710 2024-03-25T17:31:42.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:42.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.944+0000 7f22066c8640 10 monclient(hunting): authenticate will time out at 5461.226878s 2024-03-25T17:31:42.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.947+0000 7f21fffff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f22000a98f0 0x7f2200139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46308/0 (socket says 172.21.15.96:46308) 2024-03-25T17:31:42.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.947+0000 7f21fffff640 1 -- 172.21.15.96:0/1523212402 learned_addr learned my addr 172.21.15.96:0/1523212402 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:42.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.950+0000 7f21fcff9640 10 client.?.objecter ms_handle_connect 0x7f22000a98f0 2024-03-25T17:31:42.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.950+0000 7f21fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:42.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.972+0000 7f21fcff9640 10 client.?.objecter ms_handle_connect 0x7f22001430c0 2024-03-25T17:31:42.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.972+0000 7f21fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:42.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974875760 0 0) 0x7f22000b7aa0 con 0x7f22000a98f0 2024-03-25T17:31:42.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:42.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:42.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:42.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T17:31:42.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.974+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ec004eb0 con 0x7f22000a98f0 2024-03-25T17:31:42.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.994+0000 7f21fcff9640 10 client.?.objecter ms_handle_connect 0x7f220013f710 2024-03-25T17:31:42.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:42.994+0000 7f21fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:43.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234232969 0 0) 0x7f21ec004eb0 con 0x7f22000a98f0 2024-03-25T17:31:43.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22000b7aa0 con 0x7f22000a98f0 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f0001950 con 0x7f22000a98f0 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 10 monclient(hunting): dump: 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:43.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.053+0000 7f21fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:43.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956189734 0 0) 0x7f220013b3f0 con 0x7f22001430c0 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.060+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ec004eb0 con 0x7f22001430c0 2024-03-25T17:31:43.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288674821 0 0) 0x7f220013b6d0 con 0x7f220013f710 2024-03-25T17:31:43.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:43.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:43.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:43.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 10 monclient(hunting): my global_id is 4337 2024-03-25T17:31:43.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.061+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f220013b3f0 con 0x7f220013f710 2024-03-25T17:31:43.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.069+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2477068167 0 0) 0x7f21ec004eb0 con 0x7f22001430c0 2024-03-25T17:31:43.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.069+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f220013b6d0 con 0x7f22001430c0 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137192596 0 0) 0x7f220013b3f0 con 0x7f220013f710 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21ec004eb0 con 0x7f220013f710 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f4001860 con 0x7f220013f710 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): dump: 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:43.073 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f8001f00 con 0x7f22001430c0 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.072+0000 7f21fcff9640 10 monclient(hunting): dump: 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:43.074 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:43.075 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:43.075 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3774537274 0 0) 0x7f22000b7aa0 con 0x7f22000a98f0 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 >> v1:172.21.15.96:6790/0 conn(0x7f220013f710 legacy=0x7f22001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 >> v1:172.21.15.203:6789/0 conn(0x7f22001430c0 legacy=0x7f22001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 10 monclient: _finish_hunting 0 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 1 monclient: found mon.a 2024-03-25T17:31:43.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200139980 con 0x7f22000a98f0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 10 monclient: _finish_auth 0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 10 monclient: _check_auth_tickets 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f21fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 5 monclient: authenticate success, global_id 4344 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 10 monclient: _renew_subs 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220014e770 con 0x7f22000a98f0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 10 monclient: _renew_subs 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f220013baf0 con 0x7f22000a98f0 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.074+0000 7f22066c8640 1 librados: init done 2024-03-25T17:31:43.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.075+0000 7f22066c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:43.078 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.075+0000 7f22066c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:43.078 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.075+0000 7f22066c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:31:43.078 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.075+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22000b3ef0 con 0x7f22000a98f0 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.395+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f00047e0 con 0x7f22000a98f0 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f21f0004b70 con 0x7f22000a98f0 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:43.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 10 monclient: dump: 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fe7fc640 4 set_mon_vals no callback set 2024-03-25T17:31:43.398 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.396+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f21f0004d90 con 0x7f22000a98f0 2024-03-25T17:31:43.399 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.397+0000 7f21ff7fe640 10 monclient: get_auth_request con 0x7f21ec06e190 auth_method 0 2024-03-25T17:31:43.399 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.397+0000 7f21fcff9640 10 client.4344.objecter ms_handle_connect 0x7f21ec06e190 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f21f0054800 con 0x7f22000a98f0 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 10 client.4344.objecter ms_dispatch 0x7f2200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21f0050310 con 0x7f22000a98f0 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:43.403 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.402+0000 7f21fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:43.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.648+0000 7f22066c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:43.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.648+0000 7f22066c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:43.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.648+0000 7f22066c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:31:43.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.648+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22000b7c50 con 0x7f22000a98f0 2024-03-25T17:31:43.692 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.691+0000 7f21fcff9640 1 -- 172.21.15.96:0/1523212402 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f21f0052240 con 0x7f22000a98f0 2024-03-25T17:31:43.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.691+0000 7f21fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:43.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.691+0000 7f21fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:43.693 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:43.693 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:43.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 10 librados: watch_flush enter 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 10 librados: watch_flush exit 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 10 client.4344.objecter successfully canceled tick 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 >> v1:172.21.15.203:6800/3611964061 conn(0x7f21ec06e190 legacy=0x7f21ec070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 10 monclient: shutdown 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 >> v1:172.21.15.96:6789/0 conn(0x7f22000a98f0 legacy=0x7f2200139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 shutdown_connections 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.694+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 >> 172.21.15.96:0/1523212402 conn(0x7f22000b7ea0 msgr2=0x7f2200074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.695+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 shutdown_connections 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.695+0000 7f22066c8640 1 -- 172.21.15.96:0/1523212402 wait complete. 2024-03-25T17:31:43.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:43.695+0000 7f22066c8640 1 librados: shutdown 2024-03-25T17:31:43.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:50.514 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:50.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.628+0000 7fa5c52fb640 10 monclient: get_monmap_and_config 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.628+0000 7fa5c52fb640 10 monclient: build_initial_monmap 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.628+0000 7fa5c52fb640 10 monclient: monmap: 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:50.629524+0000 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:50.629524+0000 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.629 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:50.630 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:50.630 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:50.630 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.628+0000 7fa5c52fb640 1 Processor -- start 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 1 -- start start 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient: init 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c00a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:50.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:50.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c0142520 con 0x7fa5c013f710 2024-03-25T17:31:50.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00b28a0 con 0x7fa5c01430c0 2024-03-25T17:31:50.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00b2a30 con 0x7fa5c00a98f0 2024-03-25T17:31:50.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:50.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.629+0000 7fa5c52fb640 10 monclient(hunting): authenticate will time out at 5468.911767s 2024-03-25T17:31:50.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.636+0000 7fa5bf7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa5c01430c0 0x7fa5c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:35942/0 (socket says 172.21.15.96:35942) 2024-03-25T17:31:50.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.636+0000 7fa5bf7fe640 1 -- 172.21.15.96:0/373449477 learned_addr learned my addr 172.21.15.96:0/373449477 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:50.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950993940 0 0) 0x7fa5c00b28a0 con 0x7fa5c01430c0 2024-03-25T17:31:50.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 10 monclient(hunting): my global_id is 4333 2024-03-25T17:31:50.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.652+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a8003520 con 0x7fa5c01430c0 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245404533 0 0) 0x7fa5c0142520 con 0x7fa5c013f710 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 10 monclient(hunting): my global_id is 4347 2024-03-25T17:31:50.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.657+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c00b28a0 con 0x7fa5c013f710 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806971026 0 0) 0x7fa5c00b2a30 con 0x7fa5c00a98f0 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 10 monclient(hunting): my global_id is 4340 2024-03-25T17:31:50.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.664+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c0142520 con 0x7fa5c00a98f0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.667+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223256349 0 0) 0x7fa5a8003520 con 0x7fa5c01430c0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.667+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 >> v1:172.21.15.96:6790/0 conn(0x7fa5c00a98f0 legacy=0x7fa5c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.667+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 >> v1:172.21.15.96:6789/0 conn(0x7fa5c013f710 legacy=0x7fa5c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.667+0000 7fa5bdffb640 10 monclient: _finish_hunting 0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.667+0000 7fa5bdffb640 1 monclient: found mon.noname-c 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00b44c0 con 0x7fa5c01430c0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: _finish_auth 0 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: _check_auth_tickets 2024-03-25T17:31:50.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b8001a10 con 0x7fa5c01430c0 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5bdffb640 10 monclient: dump: 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:50.670 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5c52fb640 5 monclient: authenticate success, global_id 4333 2024-03-25T17:31:50.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.668+0000 7fa5c52fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:50.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5b8001b70 con 0x7fa5c01430c0 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 1 -- 172.21.15.96:0/373449477 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b8001f80 con 0x7fa5c01430c0 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5c52fb640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5c52fb640 4 set_mon_vals no callback set 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.689+0000 7fa5bdffb640 10 monclient: dump: 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.690 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:50.691 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.691 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:50.691 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:50.691 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:50.691 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 10 monclient: shutdown 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 1 -- 172.21.15.96:0/373449477 >> v1:172.21.15.203:6789/0 conn(0x7fa5c01430c0 legacy=0x7fa5c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 1 -- 172.21.15.96:0/373449477 shutdown_connections 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 1 -- 172.21.15.96:0/373449477 >> 172.21.15.96:0/373449477 conn(0x7fa5c00b7ea0 msgr2=0x7fa5c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 1 -- 172.21.15.96:0/373449477 shutdown_connections 2024-03-25T17:31:50.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 1 -- 172.21.15.96:0/373449477 wait complete. 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.690+0000 7fa5c52fb640 10 monclient: build_initial_monmap 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 10 monclient: monmap: 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:50.694 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 librados: starting msgr at 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 librados: starting objecter 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 Processor -- start 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 -- start start 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 librados: setting wanted keys 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 1 librados: calling monclient init 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.691+0000 7fa5c52fb640 10 monclient: init 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c00a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c01430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c0137c60 con 0x7fa5c01430c0 2024-03-25T17:31:50.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c0139160 con 0x7fa5c013f710 2024-03-25T17:31:50.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c01380f0 con 0x7fa5c00a98f0 2024-03-25T17:31:50.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:50.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.692+0000 7fa5c52fb640 10 monclient(hunting): authenticate will time out at 5468.974693s 2024-03-25T17:31:50.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.707+0000 7fa5bf7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa5c01430c0 0x7fa5c01e2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34574/0 (socket says 172.21.15.96:34574) 2024-03-25T17:31:50.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.707+0000 7fa5bf7fe640 1 -- 172.21.15.96:0/1318823493 learned_addr learned my addr 172.21.15.96:0/1318823493 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:50.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.707+0000 7fa5a77fe640 10 client.?.objecter ms_handle_connect 0x7fa5c01430c0 2024-03-25T17:31:50.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.707+0000 7fa5a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:50.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.712+0000 7fa5a77fe640 10 client.?.objecter ms_handle_connect 0x7fa5c00a98f0 2024-03-25T17:31:50.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.712+0000 7fa5a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:50.714 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.713+0000 7fa5a77fe640 10 client.?.objecter ms_handle_connect 0x7fa5c013f710 2024-03-25T17:31:50.714 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.713+0000 7fa5a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453852701 0 0) 0x7fa5c0137c60 con 0x7fa5c01430c0 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T17:31:50.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.722+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac004eb0 con 0x7fa5c01430c0 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146861263 0 0) 0x7fa5c0139160 con 0x7fa5c013f710 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 10 monclient(hunting): my global_id is 4336 2024-03-25T17:31:50.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.724+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c0137c60 con 0x7fa5c013f710 2024-03-25T17:31:50.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818910095 0 0) 0x7fa5c01380f0 con 0x7fa5c00a98f0 2024-03-25T17:31:50.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:50.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:50.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:50.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 10 monclient(hunting): my global_id is 4343 2024-03-25T17:31:50.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.727+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c0139160 con 0x7fa5c00a98f0 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659497655 0 0) 0x7fa5c0137c60 con 0x7fa5c013f710 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5c01380f0 con 0x7fa5c013f710 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b4001970 con 0x7fa5c013f710 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 10 monclient(hunting): dump: 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.745 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:50.746 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:50.746 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:50.746 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.744+0000 7fa5a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:50.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 656113081 0 0) 0x7fa5c0139160 con 0x7fa5c00a98f0 2024-03-25T17:31:50.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5c0137c60 con 0x7fa5c00a98f0 2024-03-25T17:31:50.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b0001830 con 0x7fa5c00a98f0 2024-03-25T17:31:50.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:50.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.749+0000 7fa5a77fe640 10 monclient(hunting): dump: 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:50.751 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66003004 0 0) 0x7fa5ac004eb0 con 0x7fa5c01430c0 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5c0139160 con 0x7fa5c01430c0 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b8001ec0 con 0x7fa5c01430c0 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.760+0000 7fa5a77fe640 10 monclient(hunting): dump: 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:50.761 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:50.762 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:50.762 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:50.762 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:50.762 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:50.764 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2820335902 0 0) 0x7fa5c0137c60 con 0x7fa5c00a98f0 2024-03-25T17:31:50.764 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 >> v1:172.21.15.203:6789/0 conn(0x7fa5c013f710 legacy=0x7fa5c0133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 >> v1:172.21.15.96:6789/0 conn(0x7fa5c01430c0 legacy=0x7fa5c01e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 1 monclient: found mon.c 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c01397f0 con 0x7fa5c00a98f0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 5 monclient: authenticate success, global_id 4343 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 10 monclient: _renew_subs 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c01db280 con 0x7fa5c00a98f0 2024-03-25T17:31:50.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 10 monclient: _renew_subs 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c0133960 con 0x7fa5c00a98f0 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.763+0000 7fa5c52fb640 1 librados: init done 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.764+0000 7fa5c52fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.764+0000 7fa5c52fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.764+0000 7fa5c52fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:50.766 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:50.764+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c0139a40 con 0x7fa5c00a98f0 2024-03-25T17:31:51.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5b0001d40 con 0x7fa5c00a98f0 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa5b00037b0 con 0x7fa5c00a98f0 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5a77fe640 10 monclient: dump: 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:51.058 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.056+0000 7fa5bd7fa640 4 set_mon_vals no callback set 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.057+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fa5b0019840 con 0x7fa5c00a98f0 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.058+0000 7fa5be7fc640 10 monclient: get_auth_request con 0x7fa5ac06e160 auth_method 0 2024-03-25T17:31:51.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.058+0000 7fa5a77fe640 10 client.4343.objecter ms_handle_connect 0x7fa5ac06e160 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fa5b0053500 con 0x7fa5c00a98f0 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 10 client.4343.objecter ms_dispatch 0x7fa5c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5b004ee40 con 0x7fa5c00a98f0 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:51.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.062+0000 7fa5a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:51.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.286+0000 7fa5c52fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:51.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.286+0000 7fa5c52fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:51.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.286+0000 7fa5c52fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:31:51.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.286+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5c00627b0 con 0x7fa5c00a98f0 2024-03-25T17:31:51.401 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.400+0000 7fa5a77fe640 1 -- 172.21.15.96:0/1318823493 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fa5b0050f40 con 0x7fa5c00a98f0 2024-03-25T17:31:51.401 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.400+0000 7fa5a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:51.401 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.400+0000 7fa5a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:51.402 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:51.402 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 10 librados: watch_flush enter 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 10 librados: watch_flush exit 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 10 client.4343.objecter successfully canceled tick 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 >> v1:172.21.15.203:6800/3611964061 conn(0x7fa5ac06e160 legacy=0x7fa5ac070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 10 monclient: shutdown 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.402+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 >> v1:172.21.15.96:6790/0 conn(0x7fa5c00a98f0 legacy=0x7fa5c01dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.403+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 shutdown_connections 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.403+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 >> 172.21.15.96:0/1318823493 conn(0x7fa5c00b7ea0 msgr2=0x7fa5c013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.403+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 shutdown_connections 2024-03-25T17:31:51.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.403+0000 7fa5c52fb640 1 -- 172.21.15.96:0/1318823493 wait complete. 2024-03-25T17:31:51.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:51.403+0000 7fa5c52fb640 1 librados: shutdown 2024-03-25T17:31:51.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:58.220 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:58.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.335+0000 7fc6a2293640 10 monclient: get_monmap_and_config 2024-03-25T17:31:58.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.335+0000 7fc6a2293640 10 monclient: build_initial_monmap 2024-03-25T17:31:58.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.335+0000 7fc6a2293640 10 monclient: monmap: 2024-03-25T17:31:58.335 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:31:58.336343+0000 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:31:58.336343+0000 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:31:58.336 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.335+0000 7fc6a2293640 1 Processor -- start 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 1 -- start start 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient: init 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-a con 0x7fc69c0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:58.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c1430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:58.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c142520 con 0x7fc69c0a98f0 2024-03-25T17:31:58.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b28a0 con 0x7fc69c13f710 2024-03-25T17:31:58.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b2a30 con 0x7fc69c1430c0 2024-03-25T17:31:58.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:58.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.336+0000 7fc6a2293640 10 monclient(hunting): authenticate will time out at 5476.618677s 2024-03-25T17:31:58.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.343+0000 7fc69bfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc69c0a98f0 0x7fc69c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58224/0 (socket says 172.21.15.96:58224) 2024-03-25T17:31:58.344 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.343+0000 7fc69bfff640 1 -- 172.21.15.96:0/3909413267 learned_addr learned my addr 172.21.15.96:0/3909413267 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117294435 0 0) 0x7fc69c142520 con 0x7fc69c0a98f0 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 10 monclient(hunting): my global_id is 4353 2024-03-25T17:31:58.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.357+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc684003520 con 0x7fc69c0a98f0 2024-03-25T17:31:58.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051206543 0 0) 0x7fc69c0b2a30 con 0x7fc69c1430c0 2024-03-25T17:31:58.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): my global_id is 4346 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c142520 con 0x7fc69c1430c0 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612832904 0 0) 0x7fc69c0b28a0 con 0x7fc69c13f710 2024-03-25T17:31:58.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 10 monclient(hunting): my global_id is 4339 2024-03-25T17:31:58.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.369+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c0b2a30 con 0x7fc69c13f710 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153420288 0 0) 0x7fc684003520 con 0x7fc69c0a98f0 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 >> v1:172.21.15.96:6790/0 conn(0x7fc69c1430c0 legacy=0x7fc69c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 >> v1:172.21.15.203:6789/0 conn(0x7fc69c13f710 legacy=0x7fc69c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: _finish_hunting 0 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 monclient: found mon.noname-a 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:58.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0b44c0 con 0x7fc69c0a98f0 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: _finish_auth 0 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: _check_auth_tickets 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc68c001690 con 0x7fc69c0a98f0 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc69affd640 10 monclient: dump: 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.375 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc6a2293640 5 monclient: authenticate success, global_id 4353 2024-03-25T17:31:58.376 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.373+0000 7fc6a2293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc68c0017f0 con 0x7fc69c0a98f0 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 1 -- 172.21.15.96:0/3909413267 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc68c001c00 con 0x7fc69c0a98f0 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc6a2293640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc6a2293640 4 set_mon_vals no callback set 2024-03-25T17:31:58.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.382+0000 7fc69affd640 10 monclient: dump: 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.384 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 10 monclient: shutdown 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 1 -- 172.21.15.96:0/3909413267 >> v1:172.21.15.96:6789/0 conn(0x7fc69c0a98f0 legacy=0x7fc69c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 1 -- 172.21.15.96:0/3909413267 shutdown_connections 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 1 -- 172.21.15.96:0/3909413267 >> 172.21.15.96:0/3909413267 conn(0x7fc69c0b7ea0 msgr2=0x7fc69c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 1 -- 172.21.15.96:0/3909413267 shutdown_connections 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.383+0000 7fc6a2293640 1 -- 172.21.15.96:0/3909413267 wait complete. 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.384+0000 7fc6a2293640 10 monclient: build_initial_monmap 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.384+0000 7fc6a2293640 10 monclient: monmap: 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:31:58.386 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.384+0000 7fc6a2293640 1 librados: starting msgr at 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.384+0000 7fc6a2293640 1 librados: starting objecter 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.385+0000 7fc6a2293640 1 Processor -- start 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.385+0000 7fc6a2293640 1 -- start start 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.385+0000 7fc6a2293640 1 librados: setting wanted keys 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.385+0000 7fc6a2293640 1 librados: calling monclient init 2024-03-25T17:31:58.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.385+0000 7fc6a2293640 10 monclient: init 2024-03-25T17:31:58.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:58.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:58.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-a con 0x7fc69c0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c13f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient(hunting): picked mon.noname-b con 0x7fc69c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b7ce0 con 0x7fc69c0a98f0 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c1319b0 con 0x7fc69c1430c0 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0b7800 con 0x7fc69c13f710 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:58.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.386+0000 7fc6a2293640 10 monclient(hunting): authenticate will time out at 5476.668181s 2024-03-25T17:31:58.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.396+0000 7fc6a0830640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc69c1430c0 0x7fc69c1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:54074/0 (socket says 172.21.15.96:54074) 2024-03-25T17:31:58.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.396+0000 7fc69bfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc69c0a98f0 0x7fc69c1dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58268/0 (socket says 172.21.15.96:58268) 2024-03-25T17:31:58.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.396+0000 7fc6a0830640 1 -- 172.21.15.96:0/1966741135 learned_addr learned my addr 172.21.15.96:0/1966741135 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:31:58.398 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.398+0000 7fc698ff9640 10 client.?.objecter ms_handle_connect 0x7fc69c1430c0 2024-03-25T17:31:58.398 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.398+0000 7fc698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:58.400 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.399+0000 7fc698ff9640 10 client.?.objecter ms_handle_connect 0x7fc69c0a98f0 2024-03-25T17:31:58.400 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.399+0000 7fc698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:58.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.403+0000 7fc698ff9640 10 client.?.objecter ms_handle_connect 0x7fc69c13f710 2024-03-25T17:31:58.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.403+0000 7fc698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:58.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874435659 0 0) 0x7fc69c0b7800 con 0x7fc69c13f710 2024-03-25T17:31:58.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 10 monclient(hunting): my global_id is 4349 2024-03-25T17:31:58.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.420+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688004eb0 con 0x7fc69c13f710 2024-03-25T17:31:58.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446662223 0 0) 0x7fc69c0b7ce0 con 0x7fc69c0a98f0 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): my global_id is 4356 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c0b7800 con 0x7fc69c0a98f0 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952005465 0 0) 0x7fc69c1319b0 con 0x7fc69c1430c0 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 10 monclient(hunting): my global_id is 4342 2024-03-25T17:31:58.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.422+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c0b7ce0 con 0x7fc69c1430c0 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904344020 0 0) 0x7fc69c0b7ce0 con 0x7fc69c1430c0 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc69c1319b0 con 0x7fc69c1430c0 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc694001bd0 con 0x7fc69c1430c0 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): dump: 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.436 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718137130 0 0) 0x7fc688004eb0 con 0x7fc69c13f710 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc69c0b7ce0 con 0x7fc69c13f710 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc6900018f0 con 0x7fc69c13f710 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.437 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.434+0000 7fc698ff9640 10 monclient(hunting): dump: 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.438 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 637854386 0 0) 0x7fc69c0b7800 con 0x7fc69c0a98f0 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc688004eb0 con 0x7fc69c0a98f0 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc68c001b00 con 0x7fc69c0a98f0 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:58.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.438+0000 7fc698ff9640 10 monclient(hunting): dump: 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.440 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.448+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3881702912 0 0) 0x7fc69c1319b0 con 0x7fc69c1430c0 2024-03-25T17:31:58.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.448+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 >> v1:172.21.15.96:6790/0 conn(0x7fc69c13f710 legacy=0x7fc69c130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:58.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.448+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 >> v1:172.21.15.96:6789/0 conn(0x7fc69c0a98f0 legacy=0x7fc69c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.448+0000 7fc698ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.448+0000 7fc698ff9640 1 monclient: found mon.b 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c136f50 con 0x7fc69c1430c0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 10 monclient: _finish_auth 0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 5 monclient: authenticate success, global_id 4342 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 10 monclient: _renew_subs 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c1350f0 con 0x7fc69c1430c0 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:58.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 10 monclient: _renew_subs 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c135c60 con 0x7fc69c1430c0 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.449+0000 7fc6a2293640 1 librados: init done 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.450+0000 7fc6a2293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.450+0000 7fc6a2293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.450+0000 7fc6a2293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:58.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.450+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c0627b0 con 0x7fc69c1430c0 2024-03-25T17:31:58.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6940020c0 con 0x7fc69c1430c0 2024-03-25T17:31:58.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:58.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc694003b30 con 0x7fc69c1430c0 2024-03-25T17:31:58.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:58.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc698ff9640 10 monclient: dump: 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:31:58.737 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:31:58.738 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.735+0000 7fc69a7fc640 4 set_mon_vals no callback set 2024-03-25T17:31:58.738 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.736+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fc694003d50 con 0x7fc69c1430c0 2024-03-25T17:31:58.738 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.736+0000 7fc69bfff640 10 monclient: get_auth_request con 0x7fc68806e0f0 auth_method 0 2024-03-25T17:31:58.738 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.737+0000 7fc698ff9640 10 client.4342.objecter ms_handle_connect 0x7fc68806e0f0 2024-03-25T17:31:58.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.741+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fc694054a00 con 0x7fc69c1430c0 2024-03-25T17:31:58.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.741+0000 7fc698ff9640 10 client.4342.objecter ms_dispatch 0x7fc69c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:58.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.741+0000 7fc698ff9640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:58.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.741+0000 7fc698ff9640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:58.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.742+0000 7fc698ff9640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T17:31:58.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.742+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc694050510 con 0x7fc69c1430c0 2024-03-25T17:31:58.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.742+0000 7fc698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:58.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.742+0000 7fc698ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:58.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.969+0000 7fc6a2293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:58.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.969+0000 7fc6a2293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:58.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.969+0000 7fc6a2293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:31:58.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:58.969+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc69c13e270 con 0x7fc69c1430c0 2024-03-25T17:31:59.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.187+0000 7fc698ff9640 1 -- 172.21.15.96:0/1966741135 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fc694052440 con 0x7fc69c1430c0 2024-03-25T17:31:59.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.187+0000 7fc698ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:59.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.187+0000 7fc698ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:59.188 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:31:59.189 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.189+0000 7fc6a2293640 10 librados: watch_flush enter 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.189+0000 7fc6a2293640 10 librados: watch_flush exit 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.189+0000 7fc6a2293640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.189+0000 7fc6a2293640 10 client.4342.objecter successfully canceled tick 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.189+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 >> v1:172.21.15.203:6800/3611964061 conn(0x7fc68806e0f0 legacy=0x7fc6880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 10 monclient: shutdown 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 >> v1:172.21.15.203:6789/0 conn(0x7fc69c1430c0 legacy=0x7fc69c1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 shutdown_connections 2024-03-25T17:31:59.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 >> 172.21.15.96:0/1966741135 conn(0x7fc69c0b7ea0 msgr2=0x7fc69c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:59.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 shutdown_connections 2024-03-25T17:31:59.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 -- 172.21.15.96:0/1966741135 wait complete. 2024-03-25T17:31:59.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:31:59.190+0000 7fc6a2293640 1 librados: shutdown 2024-03-25T17:31:59.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:00.361 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:24.588 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.378 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:24.604 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.404 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:24.631 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.421 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:24.636 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.425 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:24.652 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.435 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:24.650 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.439 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:24.665 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.462 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:24.678 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:00.467 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:24.693 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:06.007 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:06.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.119+0000 7facef681640 10 monclient: get_monmap_and_config 2024-03-25T17:32:06.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.119+0000 7facef681640 10 monclient: build_initial_monmap 2024-03-25T17:32:06.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.119+0000 7facef681640 10 monclient: monmap: 2024-03-25T17:32:06.119 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:06.120054+0000 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:06.120054+0000 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:06.120 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.120+0000 7facef681640 1 Processor -- start 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.120+0000 7facef681640 1 -- start start 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.120+0000 7facef681640 10 monclient: init 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient(hunting): picked mon.noname-a con 0x7face80a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient(hunting): picked mon.noname-c con 0x7face813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient(hunting): picked mon.noname-b con 0x7face81430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8142520 con 0x7face80a98f0 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face80b28a0 con 0x7face813f710 2024-03-25T17:32:06.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face80b2a30 con 0x7face81430c0 2024-03-25T17:32:06.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:06.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.121+0000 7facef681640 10 monclient(hunting): authenticate will time out at 5484.402452s 2024-03-25T17:32:06.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.128+0000 7faced41d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7face80a98f0 0x7face80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43602/0 (socket says 172.21.15.96:43602) 2024-03-25T17:32:06.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.128+0000 7faced41d640 1 -- 172.21.15.96:0/837910527 learned_addr learned my addr 172.21.15.96:0/837910527 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298261659 0 0) 0x7face80b28a0 con 0x7face813f710 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 10 monclient(hunting): my global_id is 4345 2024-03-25T17:32:06.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.149+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd4003520 con 0x7face813f710 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773368857 0 0) 0x7face8142520 con 0x7face80a98f0 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 10 monclient(hunting): my global_id is 4359 2024-03-25T17:32:06.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.151+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face80b28a0 con 0x7face80a98f0 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977373866 0 0) 0x7face80b2a30 con 0x7face81430c0 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 10 monclient(hunting): my global_id is 4352 2024-03-25T17:32:06.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.152+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8142520 con 0x7face81430c0 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694886852 0 0) 0x7face80b28a0 con 0x7face80a98f0 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 >> v1:172.21.15.96:6790/0 conn(0x7face81430c0 legacy=0x7face814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 >> v1:172.21.15.203:6789/0 conn(0x7face813f710 legacy=0x7face81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: _finish_hunting 0 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 monclient: found mon.noname-a 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:32:06.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face80b44c0 con 0x7face80a98f0 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: _finish_auth 0 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: _check_auth_tickets 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7face4001670 con 0x7face80a98f0 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facdffff640 10 monclient: dump: 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:06.163 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:06.164 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:06.164 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facef681640 5 monclient: authenticate success, global_id 4359 2024-03-25T17:32:06.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.162+0000 7facef681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:06.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.167+0000 7facdffff640 1 -- 172.21.15.96:0/837910527 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7face40017d0 con 0x7face80a98f0 2024-03-25T17:32:06.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.167+0000 7facdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:06.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.167+0000 7facef681640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:06.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.167+0000 7facef681640 4 set_mon_vals no callback set 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.168+0000 7facef681640 10 monclient: shutdown 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.168+0000 7facef681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.168+0000 7facef681640 1 -- 172.21.15.96:0/837910527 >> v1:172.21.15.96:6789/0 conn(0x7face80a98f0 legacy=0x7face80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.168+0000 7facef681640 1 -- 172.21.15.96:0/837910527 shutdown_connections 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.168+0000 7facef681640 1 -- 172.21.15.96:0/837910527 >> 172.21.15.96:0/837910527 conn(0x7face80b7ea0 msgr2=0x7face80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.169+0000 7facef681640 1 -- 172.21.15.96:0/837910527 shutdown_connections 2024-03-25T17:32:06.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.169+0000 7facef681640 1 -- 172.21.15.96:0/837910527 wait complete. 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.169+0000 7facef681640 10 monclient: build_initial_monmap 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.169+0000 7facef681640 10 monclient: monmap: 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:06.169 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:06.170 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:06.170 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:06.170 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 librados: starting msgr at 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 librados: starting objecter 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 Processor -- start 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 -- start start 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 librados: setting wanted keys 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 1 librados: calling monclient init 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.170+0000 7facef681640 10 monclient: init 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient(hunting): picked mon.noname-a con 0x7face80a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:06.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient(hunting): picked mon.noname-c con 0x7face813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient(hunting): picked mon.noname-b con 0x7face81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8138460 con 0x7face80a98f0 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8137c90 con 0x7face81430c0 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8139700 con 0x7face813f710 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:06.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.171+0000 7facef681640 10 monclient(hunting): authenticate will time out at 5484.452164s 2024-03-25T17:32:06.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.182+0000 7faced41d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7face80a98f0 0x7face8137580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43650/0 (socket says 172.21.15.96:43650) 2024-03-25T17:32:06.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.182+0000 7faced41d640 1 -- 172.21.15.96:0/934601709 learned_addr learned my addr 172.21.15.96:0/934601709 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:06.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.183+0000 7facddffb640 10 client.?.objecter ms_handle_connect 0x7face80a98f0 2024-03-25T17:32:06.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.183+0000 7facddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:06.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.188+0000 7facddffb640 10 client.?.objecter ms_handle_connect 0x7face813f710 2024-03-25T17:32:06.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.188+0000 7facddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:06.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.191+0000 7facddffb640 10 client.?.objecter ms_handle_connect 0x7face81430c0 2024-03-25T17:32:06.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.191+0000 7facddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:06.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559264228 0 0) 0x7face8139700 con 0x7face813f710 2024-03-25T17:32:06.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 10 monclient(hunting): my global_id is 4355 2024-03-25T17:32:06.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.202+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc004eb0 con 0x7face813f710 2024-03-25T17:32:06.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463941834 0 0) 0x7face8137c90 con 0x7face81430c0 2024-03-25T17:32:06.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 10 monclient(hunting): my global_id is 4348 2024-03-25T17:32:06.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.205+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8139700 con 0x7face81430c0 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271409273 0 0) 0x7face8138460 con 0x7face80a98f0 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 10 monclient(hunting): my global_id is 4362 2024-03-25T17:32:06.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.209+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8137c90 con 0x7face80a98f0 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126089357 0 0) 0x7faccc004eb0 con 0x7face813f710 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7face8138460 con 0x7face813f710 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7facd8001950 con 0x7face813f710 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 10 monclient(hunting): dump: 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.221 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.221+0000 7facddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391861025 0 0) 0x7face8137c90 con 0x7face80a98f0 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc004eb0 con 0x7face80a98f0 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7face4001c80 con 0x7face80a98f0 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.225+0000 7facddffb640 10 monclient(hunting): dump: 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:06.225 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:06.226 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3986120070 0 0) 0x7face8138460 con 0x7face813f710 2024-03-25T17:32:06.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 >> v1:172.21.15.203:6789/0 conn(0x7face81430c0 legacy=0x7face81e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 >> v1:172.21.15.96:6789/0 conn(0x7face80a98f0 legacy=0x7face8137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 10 monclient: _finish_hunting 0 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 1 monclient: found mon.c 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face81388f0 con 0x7face813f710 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 10 monclient: _finish_auth 0 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 10 monclient: _check_auth_tickets 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 5 monclient: authenticate success, global_id 4355 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 10 monclient: _renew_subs 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face8137eb0 con 0x7face813f710 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.230+0000 7facef681640 10 monclient: _renew_subs 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:06.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7face81d7b70 con 0x7face813f710 2024-03-25T17:32:06.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 1 librados: init done 2024-03-25T17:32:06.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:06.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:06.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:06.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.231+0000 7facef681640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face80b3ef0 con 0x7face813f710 2024-03-25T17:32:06.528 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facd8003610 con 0x7face813f710 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7facd8001d80 con 0x7face813f710 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facddffb640 10 monclient: dump: 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:06.529 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.529+0000 7facdf7fe640 4 set_mon_vals no callback set 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.530+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7facd8003ae0 con 0x7face813f710 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.530+0000 7faced41d640 10 monclient: get_auth_request con 0x7faccc06df70 auth_method 0 2024-03-25T17:32:06.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.530+0000 7facddffb640 10 client.4355.objecter ms_handle_connect 0x7faccc06df70 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7facd80535e0 con 0x7face813f710 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 10 client.4355.objecter ms_dispatch 0x7face8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facd804f0f0 con 0x7face813f710 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:06.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.536+0000 7facddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:06.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.780+0000 7facef681640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:06.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.780+0000 7facef681640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:06.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.780+0000 7facef681640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:06.780 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.780+0000 7facef681640 1 -- 172.21.15.96:0/934601709 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face80627b0 con 0x7face813f710 2024-03-25T17:32:06.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.803+0000 7facddffb640 1 -- 172.21.15.96:0/934601709 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7facd8051020 con 0x7face813f710 2024-03-25T17:32:06.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.803+0000 7facddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:06.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.803+0000 7facddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:06.803 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:06.803 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 10 librados: watch_flush enter 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 10 librados: watch_flush exit 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 10 client.4355.objecter successfully canceled tick 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 >> v1:172.21.15.203:6800/3611964061 conn(0x7faccc06df70 legacy=0x7faccc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 10 monclient: shutdown 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:06.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 >> v1:172.21.15.96:6790/0 conn(0x7face813f710 legacy=0x7face81d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:06.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 shutdown_connections 2024-03-25T17:32:06.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 >> 172.21.15.96:0/934601709 conn(0x7face80b7ea0 msgr2=0x7face8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:06.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 shutdown_connections 2024-03-25T17:32:06.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 -- 172.21.15.96:0/934601709 wait complete. 2024-03-25T17:32:06.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:06.806+0000 7facef681640 1 librados: shutdown 2024-03-25T17:32:06.817 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:13.623 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:13.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.741+0000 7fa97f214640 10 monclient: get_monmap_and_config 2024-03-25T17:32:13.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.741+0000 7fa97f214640 10 monclient: build_initial_monmap 2024-03-25T17:32:13.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.741+0000 7fa97f214640 10 monclient: monmap: 2024-03-25T17:32:13.742 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:13.743065+0000 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:13.743065+0000 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:13.743 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.742+0000 7fa97f214640 1 Processor -- start 2024-03-25T17:32:13.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.743+0000 7fa97f214640 1 -- start start 2024-03-25T17:32:13.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.743+0000 7fa97f214640 10 monclient: init 2024-03-25T17:32:13.744 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-a con 0x7fa9780a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-c con 0x7fa97813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-b con 0x7fa9781430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978142520 con 0x7fa9780a98f0 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780b28a0 con 0x7fa97813f710 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780b2a30 con 0x7fa9781430c0 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:13.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.744+0000 7fa97f214640 10 monclient(hunting): authenticate will time out at 5492.026179s 2024-03-25T17:32:13.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.753+0000 7fa97cfb0640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 0x7fa9780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48286/0 (socket says 172.21.15.96:48286) 2024-03-25T17:32:13.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.753+0000 7fa97cfb0640 1 -- 172.21.15.96:0/3842482166 learned_addr learned my addr 172.21.15.96:0/3842482166 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:13.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.757+0000 7fa97cfb0640 1 -- 172.21.15.96:0/3842482166 >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 legacy=0x7fa9780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T17:32:13.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.757+0000 7fa97cfb0640 1 -- 172.21.15.96:0/3842482166 >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 legacy=0x7fa9780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:32:13.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.757+0000 7fa97cfb0640 1 --1- 172.21.15.96:0/3842482166 >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 0x7fa9780a9cd0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640928394 0 0) 0x7fa9780b28a0 con 0x7fa97813f710 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T17:32:13.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.777+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa968003520 con 0x7fa97813f710 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501646996 0 0) 0x7fa9780b2a30 con 0x7fa9781430c0 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T17:32:13.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.782+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9780b28a0 con 0x7fa9781430c0 2024-03-25T17:32:13.793 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3232660919 0 0) 0x7fa968003520 con 0x7fa97813f710 2024-03-25T17:32:13.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 >> v1:172.21.15.96:6790/0 conn(0x7fa9781430c0 legacy=0x7fa97814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:13.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 legacy=0x7fa9780a9cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:32:13.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: _finish_hunting 0 2024-03-25T17:32:13.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 monclient: found mon.noname-c 2024-03-25T17:32:13.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780b44c0 con 0x7fa97813f710 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: _finish_auth 0 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: _check_auth_tickets 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa96c001690 con 0x7fa97813f710 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa97f214640 5 monclient: authenticate success, global_id 4351 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa97f214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa9777fe640 10 monclient: dump: 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:13.806 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.792+0000 7fa97f214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa96c0017f0 con 0x7fa97813f710 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 1 -- 172.21.15.96:0/3842482166 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa96c001c00 con 0x7fa97813f710 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa97f214640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa97f214640 4 set_mon_vals no callback set 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.831+0000 7fa9777fe640 10 monclient: dump: 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.832 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:13.833 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 10 monclient: shutdown 2024-03-25T17:32:13.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:13.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 1 -- 172.21.15.96:0/3842482166 >> v1:172.21.15.203:6789/0 conn(0x7fa97813f710 legacy=0x7fa9781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:13.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 1 -- 172.21.15.96:0/3842482166 shutdown_connections 2024-03-25T17:32:13.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 1 -- 172.21.15.96:0/3842482166 >> 172.21.15.96:0/3842482166 conn(0x7fa9780b7ea0 msgr2=0x7fa9780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 1 -- 172.21.15.96:0/3842482166 shutdown_connections 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 1 -- 172.21.15.96:0/3842482166 wait complete. 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 10 monclient: build_initial_monmap 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.832+0000 7fa97f214640 10 monclient: monmap: 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:13.835 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 librados: starting msgr at 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 librados: starting objecter 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 Processor -- start 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 -- start start 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 librados: setting wanted keys 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 1 librados: calling monclient init 2024-03-25T17:32:13.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.833+0000 7fa97f214640 10 monclient: init 2024-03-25T17:32:13.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:13.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:13.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-a con 0x7fa9780a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:13.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-c con 0x7fa97813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:13.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient(hunting): picked mon.noname-b con 0x7fa9781430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:13.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978139330 con 0x7fa9780a98f0 2024-03-25T17:32:13.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780abe70 con 0x7fa9781430c0 2024-03-25T17:32:13.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780ac150 con 0x7fa97813f710 2024-03-25T17:32:13.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:13.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.834+0000 7fa97f214640 10 monclient(hunting): authenticate will time out at 5492.116926s 2024-03-25T17:32:13.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.840+0000 7fa97d7b1640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa9781430c0 0x7fa9781e2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:56920/0 (socket says 172.21.15.96:56920) 2024-03-25T17:32:13.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.840+0000 7fa97d7b1640 1 -- 172.21.15.96:0/1198056452 learned_addr learned my addr 172.21.15.96:0/1198056452 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:13.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.841+0000 7fa9757fa640 10 client.?.objecter ms_handle_connect 0x7fa9781430c0 2024-03-25T17:32:13.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.841+0000 7fa9757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.842+0000 7fa9757fa640 10 client.?.objecter ms_handle_connect 0x7fa9780a98f0 2024-03-25T17:32:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.842+0000 7fa9757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:13.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.846+0000 7fa9757fa640 10 client.?.objecter ms_handle_connect 0x7fa97813f710 2024-03-25T17:32:13.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.846+0000 7fa9757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237090659 0 0) 0x7fa9780abe70 con 0x7fa9781430c0 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T17:32:13.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.848+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa95c004eb0 con 0x7fa9781430c0 2024-03-25T17:32:13.851 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497369773 0 0) 0x7fa978139330 con 0x7fa9780a98f0 2024-03-25T17:32:13.851 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:13.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:13.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:13.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T17:32:13.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.851+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9780abe70 con 0x7fa9780a98f0 2024-03-25T17:32:13.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426959088 0 0) 0x7fa9780ac150 con 0x7fa97813f710 2024-03-25T17:32:13.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:13.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:13.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:13.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 10 monclient(hunting): my global_id is 4361 2024-03-25T17:32:13.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.859+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa978139330 con 0x7fa97813f710 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932440208 0 0) 0x7fa95c004eb0 con 0x7fa9781430c0 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9780ac150 con 0x7fa9781430c0 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa970001c70 con 0x7fa9781430c0 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 10 monclient(hunting): dump: 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.864 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.865 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.863+0000 7fa9757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957524648 0 0) 0x7fa9780abe70 con 0x7fa9780a98f0 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa95c004eb0 con 0x7fa9780a98f0 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa9640014d0 con 0x7fa9780a98f0 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.873+0000 7fa9757fa640 10 monclient(hunting): dump: 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:13.874 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:13.875 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:13.875 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:13.875 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:13.875 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:13.875 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3553564371 0 0) 0x7fa9780ac150 con 0x7fa9781430c0 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 >> v1:172.21.15.96:6790/0 conn(0x7fa97813f710 legacy=0x7fa9780ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 >> v1:172.21.15.96:6789/0 conn(0x7fa9780a98f0 legacy=0x7fa9781389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 10 monclient: _finish_hunting 0 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 1 monclient: found mon.b 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:13.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780acc80 con 0x7fa9781430c0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 10 monclient: _finish_auth 0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 10 monclient: _check_auth_tickets 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa9757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 5 monclient: authenticate success, global_id 4354 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 10 monclient: _renew_subs 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa978137e30 con 0x7fa9781430c0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 10 monclient: _renew_subs 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa978136880 con 0x7fa9781430c0 2024-03-25T17:32:13.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.877+0000 7fa97f214640 1 librados: init done 2024-03-25T17:32:13.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.878+0000 7fa97f214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:13.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.878+0000 7fa97f214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:13.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.878+0000 7fa97f214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:13.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:13.878+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9780627b0 con 0x7fa9781430c0 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa970003990 con 0x7fa9781430c0 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa970003d60 con 0x7fa9781430c0 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa9757fa640 10 monclient: dump: 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:14.163 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:14.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.162+0000 7fa976ffd640 4 set_mon_vals no callback set 2024-03-25T17:32:14.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.163+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fa970003f80 con 0x7fa9781430c0 2024-03-25T17:32:14.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.164+0000 7fa97cfb0640 10 monclient: get_auth_request con 0x7fa95c06df70 auth_method 0 2024-03-25T17:32:14.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.164+0000 7fa9757fa640 10 client.4354.objecter ms_handle_connect 0x7fa95c06df70 2024-03-25T17:32:14.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fa970054ad0 con 0x7fa9781430c0 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 10 client.4354.objecter ms_dispatch 0x7fa978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9700505e0 con 0x7fa9781430c0 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:14.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.168+0000 7fa9757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:14.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.409+0000 7fa97f214640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:14.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.409+0000 7fa97f214640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:14.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.409+0000 7fa97f214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:14.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.409+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9781390e0 con 0x7fa9781430c0 2024-03-25T17:32:14.432 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.431+0000 7fa9757fa640 1 -- 172.21.15.96:0/1198056452 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fa970052510 con 0x7fa9781430c0 2024-03-25T17:32:14.432 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.431+0000 7fa9757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:14.432 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.431+0000 7fa9757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:14.433 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:14.433 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:14.435 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 10 librados: watch_flush enter 2024-03-25T17:32:14.435 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 10 librados: watch_flush exit 2024-03-25T17:32:14.435 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T17:32:14.435 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 10 client.4354.objecter successfully canceled tick 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 >> v1:172.21.15.203:6800/3611964061 conn(0x7fa95c06df70 legacy=0x7fa95c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 10 monclient: shutdown 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 >> v1:172.21.15.203:6789/0 conn(0x7fa9781430c0 legacy=0x7fa9781e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 shutdown_connections 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.434+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 >> 172.21.15.96:0/1198056452 conn(0x7fa9780b7ea0 msgr2=0x7fa978074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.435+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 shutdown_connections 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.435+0000 7fa97f214640 1 -- 172.21.15.96:0/1198056452 wait complete. 2024-03-25T17:32:14.436 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:14.435+0000 7fa97f214640 1 librados: shutdown 2024-03-25T17:32:14.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:20.416 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:44.643 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.428 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:44.655 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.461 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:03:44.687 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.473 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:44.688 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.480 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:44.696 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.487 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:44.713 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.491 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:44.718 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.523 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:44.738 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:20.529 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:03:44.755 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:21.255 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:21.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.369+0000 7f7ad738f640 10 monclient: get_monmap_and_config 2024-03-25T17:32:21.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.369+0000 7f7ad738f640 10 monclient: build_initial_monmap 2024-03-25T17:32:21.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.369+0000 7f7ad738f640 10 monclient: monmap: 2024-03-25T17:32:21.369 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:21.370307+0000 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:21.370307+0000 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:21.370 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.369+0000 7f7ad738f640 1 Processor -- start 2024-03-25T17:32:21.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 1 -- start start 2024-03-25T17:32:21.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient: init 2024-03-25T17:32:21.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:21.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad00a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad01430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.370+0000 7f7ad738f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad0142520 con 0x7f7ad00a98f0 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.371+0000 7f7ad738f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad00b28a0 con 0x7f7ad013f710 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.371+0000 7f7ad738f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad00b2a30 con 0x7f7ad01430c0 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.371+0000 7f7ad738f640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:21.372 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.371+0000 7f7ad738f640 10 monclient(hunting): authenticate will time out at 5499.653082s 2024-03-25T17:32:21.376 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.375+0000 7f7ad592c640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f7ad01430c0 0x7f7ad014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:53910/0 (socket says 172.21.15.96:53910) 2024-03-25T17:32:21.376 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.375+0000 7f7ad592c640 1 -- 172.21.15.96:0/4028492688 learned_addr learned my addr 172.21.15.96:0/4028492688 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:21.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842120532 0 0) 0x7f7ad00b2a30 con 0x7f7ad01430c0 2024-03-25T17:32:21.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:21.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:21.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:21.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T17:32:21.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.385+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7abc003520 con 0x7f7ad01430c0 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013593010 0 0) 0x7f7ad0142520 con 0x7f7ad00a98f0 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 10 monclient(hunting): my global_id is 4368 2024-03-25T17:32:21.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.401+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad00b2a30 con 0x7f7ad00a98f0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683396164 0 0) 0x7f7ad00b2a30 con 0x7f7ad00a98f0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 >> v1:172.21.15.96:6790/0 conn(0x7f7ad01430c0 legacy=0x7f7ad014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 >> v1:172.21.15.203:6789/0 conn(0x7f7ad013f710 legacy=0x7f7ad01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 monclient: found mon.noname-a 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad00b44c0 con 0x7f7ad00a98f0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: _finish_auth 0 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:21.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7acc001670 con 0x7f7ad00a98f0 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ad738f640 5 monclient: authenticate success, global_id 4368 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ad738f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ac7fff640 10 monclient: dump: 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:21.475 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.476 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.472+0000 7f7ad738f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:21.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7acc001890 con 0x7f7ad00a98f0 2024-03-25T17:32:21.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:21.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 1 -- 172.21.15.96:0/4028492688 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7acc001ba0 con 0x7f7ad00a98f0 2024-03-25T17:32:21.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:21.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.491+0000 7f7ac7fff640 10 monclient: dump: 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.492+0000 7f7ad738f640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:21.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.492+0000 7f7ad738f640 4 set_mon_vals no callback set 2024-03-25T17:32:21.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.492+0000 7f7ad738f640 10 monclient: shutdown 2024-03-25T17:32:21.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.492+0000 7f7ad738f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:21.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.492+0000 7f7ad738f640 1 -- 172.21.15.96:0/4028492688 >> v1:172.21.15.96:6789/0 conn(0x7f7ad00a98f0 legacy=0x7f7ad00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:21.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.493+0000 7f7ad738f640 1 -- 172.21.15.96:0/4028492688 shutdown_connections 2024-03-25T17:32:21.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.493+0000 7f7ad738f640 1 -- 172.21.15.96:0/4028492688 >> 172.21.15.96:0/4028492688 conn(0x7f7ad00b7ea0 msgr2=0x7f7ad00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.493+0000 7f7ad738f640 1 -- 172.21.15.96:0/4028492688 shutdown_connections 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.493+0000 7f7ad738f640 1 -- 172.21.15.96:0/4028492688 wait complete. 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 10 monclient: build_initial_monmap 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 10 monclient: monmap: 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:21.495 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 librados: starting msgr at 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 librados: starting objecter 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 Processor -- start 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 -- start start 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 librados: setting wanted keys 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 1 librados: calling monclient init 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.494+0000 7f7ad738f640 10 monclient: init 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad00a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad01daa20 con 0x7f7ad013f710 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad01371e0 con 0x7f7ad01430c0 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad01374c0 con 0x7f7ad00a98f0 2024-03-25T17:32:21.497 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:21.498 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.495+0000 7f7ad738f640 10 monclient(hunting): authenticate will time out at 5499.777921s 2024-03-25T17:32:21.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.519+0000 7f7ad592c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7ad01430c0 0x7f7ad01e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:60422/0 (socket says 172.21.15.96:60422) 2024-03-25T17:32:21.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.519+0000 7f7ad592c640 1 -- 172.21.15.96:0/2501640046 learned_addr learned my addr 172.21.15.96:0/2501640046 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:21.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.521+0000 7f7ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f7ad01430c0 2024-03-25T17:32:21.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.521+0000 7f7ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:21.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.541+0000 7f7ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f7ad013f710 2024-03-25T17:32:21.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.541+0000 7f7ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:21.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.543+0000 7f7ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f7ad00a98f0 2024-03-25T17:32:21.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.543+0000 7f7ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544933565 0 0) 0x7f7ad01daa20 con 0x7f7ad013f710 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 10 monclient(hunting): my global_id is 4371 2024-03-25T17:32:21.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.547+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab4004eb0 con 0x7f7ad013f710 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1447622818 0 0) 0x7f7ab4004eb0 con 0x7f7ad013f710 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ad01daa20 con 0x7f7ad013f710 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7ac00018c0 con 0x7f7ad013f710 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 10 monclient(hunting): dump: 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:21.576 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:21.577 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.575+0000 7f7ac5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.641+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838675005 0 0) 0x7f7ad01daa20 con 0x7f7ad013f710 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 >> v1:172.21.15.96:6790/0 conn(0x7f7ad00a98f0 legacy=0x7f7ad01352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 >> v1:172.21.15.203:6789/0 conn(0x7f7ad01430c0 legacy=0x7f7ad01e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 1 monclient: found mon.a 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad013c040 con 0x7f7ad013f710 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:21.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 5 monclient: authenticate success, global_id 4371 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 10 monclient: _renew_subs 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad01359f0 con 0x7f7ad013f710 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 10 monclient: _renew_subs 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad01d75d0 con 0x7f7ad013f710 2024-03-25T17:32:21.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.642+0000 7f7ad738f640 1 librados: init done 2024-03-25T17:32:21.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.643+0000 7f7ad738f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:21.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.643+0000 7f7ad738f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:21.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.643+0000 7f7ad738f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:21.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.643+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad00627b0 con 0x7f7ad013f710 2024-03-25T17:32:21.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ac00035f0 con 0x7f7ad013f710 2024-03-25T17:32:21.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:21.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7ac00039c0 con 0x7f7ad013f710 2024-03-25T17:32:21.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:21.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac5ffb640 10 monclient: dump: 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:21.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.922+0000 7f7ac77fe640 4 set_mon_vals no callback set 2024-03-25T17:32:21.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.923+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7ac0003be0 con 0x7f7ad013f710 2024-03-25T17:32:21.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.924+0000 7f7ad512b640 10 monclient: get_auth_request con 0x7f7ab406dce0 auth_method 0 2024-03-25T17:32:21.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.924+0000 7f7ac5ffb640 10 client.4371.objecter ms_handle_connect 0x7f7ab406dce0 2024-03-25T17:32:21.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f7ac0053550 con 0x7f7ad013f710 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 10 client.4371.objecter ms_dispatch 0x7f7ad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ac004f060 con 0x7f7ad013f710 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:21.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:21.928+0000 7f7ac5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:22.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.180+0000 7f7ad738f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:22.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.180+0000 7f7ad738f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:22.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.180+0000 7f7ad738f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:22.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.180+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ad01dabb0 con 0x7f7ad013f710 2024-03-25T17:32:22.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.215+0000 7f7ac5ffb640 1 -- 172.21.15.96:0/2501640046 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f7ac0050f90 con 0x7f7ad013f710 2024-03-25T17:32:22.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.215+0000 7f7ac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:22.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.215+0000 7f7ac5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:22.216 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:22.216 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 10 librados: watch_flush enter 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 10 librados: watch_flush exit 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 10 client.4371.objecter successfully canceled tick 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 >> v1:172.21.15.203:6800/3611964061 conn(0x7f7ab406dce0 legacy=0x7f7ab40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:22.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 10 monclient: shutdown 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 >> v1:172.21.15.96:6789/0 conn(0x7f7ad013f710 legacy=0x7f7ad01daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 shutdown_connections 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 >> 172.21.15.96:0/2501640046 conn(0x7f7ad00b7ea0 msgr2=0x7f7ad0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 shutdown_connections 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 -- 172.21.15.96:0/2501640046 wait complete. 2024-03-25T17:32:22.220 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:22.218+0000 7f7ad738f640 1 librados: shutdown 2024-03-25T17:32:22.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:29.037 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:29.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 10 monclient: get_monmap_and_config 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 10 monclient: build_initial_monmap 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 10 monclient: monmap: 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:29.142445+0000 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:29.142445+0000 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:29.142 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 1 Processor -- start 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 1 -- start start 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.141+0000 7f59a3e98640 10 monclient: init 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-c con 0x7f599c0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-a con 0x7f599c0b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-b con 0x7f599c14be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0af940 con 0x7f599c0b1730 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0b0a70 con 0x7f599c0b4d60 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0b0d50 con 0x7f599c14be90 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:29.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.142+0000 7f59a3e98640 10 monclient(hunting): authenticate will time out at 5507.424482s 2024-03-25T17:32:29.160 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.159+0000 7f59a1c34640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f599c0b4d60 0x7f599c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:56252/0 (socket says 172.21.15.96:56252) 2024-03-25T17:32:29.160 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.159+0000 7f59a1c34640 1 -- 172.21.15.96:0/985634407 learned_addr learned my addr 172.21.15.96:0/985634407 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957142496 0 0) 0x7f599c0b0d50 con 0x7f599c14be90 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 10 monclient(hunting): _negotiate 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 10 monclient(hunting): my global_id is 4367 2024-03-25T17:32:29.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.175+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003520 con 0x7f599c14be90 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257049185 0 0) 0x7f599c0b0a70 con 0x7f599c0b4d60 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 10 monclient(hunting): _negotiate 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 10 monclient(hunting): my global_id is 4363 2024-03-25T17:32:29.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.182+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c0b0d50 con 0x7f599c0b4d60 2024-03-25T17:32:29.197 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608763242 0 0) 0x7f5988003520 con 0x7f599c14be90 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 >> v1:172.21.15.203:6789/0 conn(0x7f599c0b4d60 legacy=0x7f599c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 >> v1:172.21.15.96:6789/0 conn(0x7f599c0b1730 legacy=0x7f599c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: _finish_hunting 0 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 monclient: found mon.noname-b 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0b1030 con 0x7f599c14be90 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: _finish_auth 0 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: _check_auth_tickets 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5994001a10 con 0x7f599c14be90 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:29.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a0c32640 10 monclient: dump: 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a3e98640 5 monclient: authenticate success, global_id 4367 2024-03-25T17:32:29.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.197+0000 7f59a3e98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5994001b70 con 0x7f599c14be90 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 1 -- 172.21.15.96:0/985634407 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5994001f80 con 0x7f599c14be90 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a0c32640 10 monclient: dump: 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:29.221 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a3e98640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:29.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.220+0000 7f59a3e98640 4 set_mon_vals no callback set 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.221+0000 7f59a3e98640 10 monclient: shutdown 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.221+0000 7f59a3e98640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.221+0000 7f59a3e98640 1 -- 172.21.15.96:0/985634407 >> v1:172.21.15.96:6790/0 conn(0x7f599c14be90 legacy=0x7f599c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.221+0000 7f59a3e98640 1 -- 172.21.15.96:0/985634407 shutdown_connections 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.221+0000 7f59a3e98640 1 -- 172.21.15.96:0/985634407 >> 172.21.15.96:0/985634407 conn(0x7f599c0617a0 msgr2=0x7f599c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.222+0000 7f59a3e98640 1 -- 172.21.15.96:0/985634407 shutdown_connections 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.222+0000 7f59a3e98640 1 -- 172.21.15.96:0/985634407 wait complete. 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.222+0000 7f59a3e98640 10 monclient: build_initial_monmap 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.222+0000 7f59a3e98640 10 monclient: monmap: 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:29.224 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 librados: starting msgr at 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 librados: starting objecter 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 Processor -- start 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 -- start start 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 librados: setting wanted keys 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 1 librados: calling monclient init 2024-03-25T17:32:29.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.223+0000 7f59a3e98640 10 monclient: init 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-c con 0x7f599c0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-a con 0x7f599c0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient(hunting): picked mon.noname-b con 0x7f599c14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:29.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c137b10 con 0x7f599c0b4d60 2024-03-25T17:32:29.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c1377e0 con 0x7f599c14be90 2024-03-25T17:32:29.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c137fa0 con 0x7f599c0b1730 2024-03-25T17:32:29.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:29.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.224+0000 7f59a3e98640 10 monclient(hunting): authenticate will time out at 5507.506340s 2024-03-25T17:32:29.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.235+0000 7f59a1433640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f599c0b4d60 0x7f599c1daed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43152/0 (socket says 172.21.15.96:43152) 2024-03-25T17:32:29.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.235+0000 7f59a1433640 1 -- 172.21.15.96:0/3851571213 learned_addr learned my addr 172.21.15.96:0/3851571213 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:29.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.237+0000 7f59927fc640 10 client.?.objecter ms_handle_connect 0x7f599c0b4d60 2024-03-25T17:32:29.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.237+0000 7f59927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59a1c34640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b1730 legacy=0x7f599c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59a1c34640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b1730 legacy=0x7f599c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59a1c34640 1 --1- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b1730 0x7f599c134eb0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59a1c34640 1 --1- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b1730 0x7f599c134eb0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59927fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.96:6790/0 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59927fc640 10 client.?.objecter ms_handle_connect 0x7f599c14be90 2024-03-25T17:32:29.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.242+0000 7f59927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611756220 0 0) 0x7f599c1377e0 con 0x7f599c14be90 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T17:32:29.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.249+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5980004eb0 con 0x7f599c14be90 2024-03-25T17:32:29.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835780108 0 0) 0x7f599c137b10 con 0x7f599c0b4d60 2024-03-25T17:32:29.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:29.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:29.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:29.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 10 monclient(hunting): my global_id is 4377 2024-03-25T17:32:29.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.252+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c1377e0 con 0x7f599c0b4d60 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3217371017 0 0) 0x7f5980004eb0 con 0x7f599c14be90 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f599c137b10 con 0x7f599c14be90 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5994001e80 con 0x7f599c14be90 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:29.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 10 monclient(hunting): dump: 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.263 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.261+0000 7f59927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485483191 0 0) 0x7f599c1377e0 con 0x7f599c0b4d60 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5980004eb0 con 0x7f599c0b4d60 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f598c0019b0 con 0x7f599c0b4d60 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.276+0000 7f59927fc640 10 monclient(hunting): dump: 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:29.277 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:29.278 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 138864145 0 0) 0x7f599c137b10 con 0x7f599c14be90 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6790/0 conn(0x7f599c0b1730 legacy=0x7f599c134eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.96:6789/0 conn(0x7f599c0b4d60 legacy=0x7f599c1daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 10 monclient: _finish_hunting 0 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 1 monclient: found mon.b 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c142030 con 0x7f599c14be90 2024-03-25T17:32:29.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 10 monclient: _finish_auth 0 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 10 monclient: _check_auth_tickets 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 5 monclient: authenticate success, global_id 4366 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 10 monclient: _renew_subs 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c140840 con 0x7f599c14be90 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 10 monclient: _renew_subs 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c1357a0 con 0x7f599c14be90 2024-03-25T17:32:29.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.278+0000 7f59a3e98640 1 librados: init done 2024-03-25T17:32:29.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.279+0000 7f59a3e98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:29.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.279+0000 7f59a3e98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:29.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.279+0000 7f59a3e98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:29.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.279+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c0627b0 con 0x7f599c14be90 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5994002040 con 0x7f599c14be90 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5994003dc0 con 0x7f599c14be90 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f59927fc640 10 monclient: dump: 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:29.566 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:29.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.565+0000 7f5993fff640 4 set_mon_vals no callback set 2024-03-25T17:32:29.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.566+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5994019d90 con 0x7f599c14be90 2024-03-25T17:32:29.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.566+0000 7f59a1c34640 10 monclient: get_auth_request con 0x7f598006de90 auth_method 0 2024-03-25T17:32:29.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.566+0000 7f59927fc640 10 client.4366.objecter ms_handle_connect 0x7f598006de90 2024-03-25T17:32:29.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5994053800 con 0x7f599c14be90 2024-03-25T17:32:29.571 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 10 client.4366.objecter ms_dispatch 0x7f599c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f599404f310 con 0x7f599c14be90 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:29.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.571+0000 7f59927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:29.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.827+0000 7f59a3e98640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:29.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.828+0000 7f59a3e98640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:29.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.828+0000 7f59a3e98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:29.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.828+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f599c0b75b0 con 0x7f599c14be90 2024-03-25T17:32:29.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.872+0000 7f59927fc640 1 -- 172.21.15.96:0/3851571213 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5994051240 con 0x7f599c14be90 2024-03-25T17:32:29.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.872+0000 7f59927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:29.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.872+0000 7f59927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:29.873 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:29.873 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:29.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 10 librados: watch_flush enter 2024-03-25T17:32:29.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 10 librados: watch_flush exit 2024-03-25T17:32:29.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T17:32:29.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 10 client.4366.objecter successfully canceled tick 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.203:6800/3611964061 conn(0x7f598006de90 legacy=0x7f5980070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 10 monclient: shutdown 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 >> v1:172.21.15.203:6789/0 conn(0x7f599c14be90 legacy=0x7f599c1e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 shutdown_connections 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 >> 172.21.15.96:0/3851571213 conn(0x7f599c0617a0 msgr2=0x7f599c13c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 shutdown_connections 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 -- 172.21.15.96:0/3851571213 wait complete. 2024-03-25T17:32:29.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:29.874+0000 7f59a3e98640 1 librados: shutdown 2024-03-25T17:32:29.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:36.694 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:36.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.803+0000 7fcfd4a0f640 10 monclient: get_monmap_and_config 2024-03-25T17:32:36.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.803+0000 7fcfd4a0f640 10 monclient: build_initial_monmap 2024-03-25T17:32:36.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.803+0000 7fcfd4a0f640 10 monclient: monmap: 2024-03-25T17:32:36.806 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:36.806 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:36.806 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:36.805099+0000 2024-03-25T17:32:36.806 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:36.805099+0000 2024-03-25T17:32:36.806 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:36.807 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:36.807 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:36.807 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:36.807 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:36.807 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:36.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.804+0000 7fcfd4a0f640 1 Processor -- start 2024-03-25T17:32:36.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.804+0000 7fcfd4a0f640 1 -- start start 2024-03-25T17:32:36.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.804+0000 7fcfd4a0f640 10 monclient: init 2024-03-25T17:32:36.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:36.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:32:36.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfd00a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:36.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:36.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfd01430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:36.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0142520 con 0x7fcfd01430c0 2024-03-25T17:32:36.812 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd00b28a0 con 0x7fcfd013f710 2024-03-25T17:32:36.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd00b2a30 con 0x7fcfd00a98f0 2024-03-25T17:32:36.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:36.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.805+0000 7fcfd4a0f640 10 monclient(hunting): authenticate will time out at 5515.087840s 2024-03-25T17:32:36.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.809+0000 7fcfce59c640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fcfd00a98f0 0x7fcfd00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:53166/0 (socket says 172.21.15.96:53166) 2024-03-25T17:32:36.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.809+0000 7fcfce59c640 1 -- 172.21.15.96:0/2625944636 learned_addr learned my addr 172.21.15.96:0/2625944636 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:36.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713656127 0 0) 0x7fcfd0142520 con 0x7fcfd01430c0 2024-03-25T17:32:36.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:36.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:36.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:36.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 10 monclient(hunting): my global_id is 4380 2024-03-25T17:32:36.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.840+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfbc003520 con 0x7fcfd01430c0 2024-03-25T17:32:36.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3802878930 0 0) 0x7fcfbc003520 con 0x7fcfd01430c0 2024-03-25T17:32:36.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 >> v1:172.21.15.96:6790/0 conn(0x7fcfd00a98f0 legacy=0x7fcfd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:36.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 >> v1:172.21.15.203:6789/0 conn(0x7fcfd013f710 legacy=0x7fcfd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:36.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: _finish_hunting 0 2024-03-25T17:32:36.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 monclient: found mon.noname-a 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00b44c0 con 0x7fcfd01430c0 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: _finish_auth 0 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: _check_auth_tickets 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:36.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcfc8001a10 con 0x7fcfd01430c0 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfd4a0f640 5 monclient: authenticate success, global_id 4380 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfd4a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfcdd9b640 10 monclient: dump: 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:36.929 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:36.930 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.930 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.930 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:36.930 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:36.930 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:36.931 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:36.931 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:36.931 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:36.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.925+0000 7fcfd4a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:36.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfc8001b70 con 0x7fcfd01430c0 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 1 -- 172.21.15.96:0/2625944636 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcfc8001f80 con 0x7fcfd01430c0 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 4 set_mon_vals no callback set 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfcdd9b640 10 monclient: dump: 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.948 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:36.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 10 monclient: shutdown 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2625944636 >> v1:172.21.15.96:6789/0 conn(0x7fcfd01430c0 legacy=0x7fcfd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2625944636 shutdown_connections 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2625944636 >> 172.21.15.96:0/2625944636 conn(0x7fcfd00b7ea0 msgr2=0x7fcfd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2625944636 shutdown_connections 2024-03-25T17:32:36.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.942+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2625944636 wait complete. 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.943+0000 7fcfd4a0f640 10 monclient: build_initial_monmap 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.943+0000 7fcfd4a0f640 10 monclient: monmap: 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:36.951 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 librados: starting msgr at 2024-03-25T17:32:36.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 librados: starting objecter 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 Processor -- start 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 -- start start 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 librados: setting wanted keys 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 librados: calling monclient init 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient: init 2024-03-25T17:32:36.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd00a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfd013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfd01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0137f10 con 0x7fcfd013f710 2024-03-25T17:32:36.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd01dc070 con 0x7fcfd01430c0 2024-03-25T17:32:36.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0137100 con 0x7fcfd00a98f0 2024-03-25T17:32:36.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:36.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.944+0000 7fcfd4a0f640 10 monclient(hunting): authenticate will time out at 5515.226716s 2024-03-25T17:32:36.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.952+0000 7fcfc7fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcfd013f710 0x7fcfd01db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35186/0 (socket says 172.21.15.96:35186) 2024-03-25T17:32:36.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.952+0000 7fcfc7fff640 1 -- 172.21.15.96:0/2614491159 learned_addr learned my addr 172.21.15.96:0/2614491159 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:36.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.955+0000 7fcfc6ffd640 10 client.?.objecter ms_handle_connect 0x7fcfd013f710 2024-03-25T17:32:36.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.955+0000 7fcfc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:36.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478156485 0 0) 0x7fcfd0137f10 con 0x7fcfd013f710 2024-03-25T17:32:36.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:36.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:36.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:36.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T17:32:36.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.962+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb0004eb0 con 0x7fcfd013f710 2024-03-25T17:32:36.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.963+0000 7fcfc6ffd640 10 client.?.objecter ms_handle_connect 0x7fcfd01430c0 2024-03-25T17:32:36.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.963+0000 7fcfc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:36.983 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970841519 0 0) 0x7fcfb0004eb0 con 0x7fcfd013f710 2024-03-25T17:32:36.983 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfd0137f10 con 0x7fcfd013f710 2024-03-25T17:32:36.983 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcfb80019d0 con 0x7fcfd013f710 2024-03-25T17:32:36.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:36.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:36.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:36.984 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:36.984 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:36.985 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.985 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:36.985 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:36.985 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:36.985 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:36.986 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:36.986 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:36.986 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:36.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.981+0000 7fcfc6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:36.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2700676770 0 0) 0x7fcfd0137f10 con 0x7fcfd013f710 2024-03-25T17:32:36.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 >> v1:172.21.15.96:6790/0 conn(0x7fcfd00a98f0 legacy=0x7fcfd0137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 >> v1:172.21.15.203:6789/0 conn(0x7fcfd01430c0 legacy=0x7fcfd01db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 1 monclient: found mon.a 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:36.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd01373e0 con 0x7fcfd013f710 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 5 monclient: authenticate success, global_id 4383 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 10 monclient: _renew_subs 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:36.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd01358e0 con 0x7fcfd013f710 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 10 monclient: _renew_subs 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd0136150 con 0x7fcfd013f710 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.988+0000 7fcfd4a0f640 1 librados: init done 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.989+0000 7fcfd4a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:36.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.989+0000 7fcfd4a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:36.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.989+0000 7fcfd4a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:36.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:36.989+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd00627b0 con 0x7fcfd013f710 2024-03-25T17:32:37.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfb8001c90 con 0x7fcfd013f710 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fcfb8003a60 con 0x7fcfd013f710 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfc6ffd640 10 monclient: dump: 2024-03-25T17:32:37.247 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:37.248 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:37.249 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:37.249 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:37.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.245+0000 7fcfcd59a640 4 set_mon_vals no callback set 2024-03-25T17:32:37.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.246+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fcfb8003c80 con 0x7fcfd013f710 2024-03-25T17:32:37.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.247+0000 7fcfce59c640 10 monclient: get_auth_request con 0x7fcfb006dc70 auth_method 0 2024-03-25T17:32:37.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.247+0000 7fcfc6ffd640 10 client.4383.objecter ms_handle_connect 0x7fcfb006dc70 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.250+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fcfb8054870 con 0x7fcfd013f710 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.250+0000 7fcfc6ffd640 10 client.4383.objecter ms_dispatch 0x7fcfd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.250+0000 7fcfc6ffd640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.250+0000 7fcfc6ffd640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.251+0000 7fcfc6ffd640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-25T17:32:37.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.251+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcfb8050380 con 0x7fcfd013f710 2024-03-25T17:32:37.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.251+0000 7fcfc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:37.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.251+0000 7fcfc6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:37.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.502+0000 7fcfd4a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:37.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.502+0000 7fcfd4a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:37.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.502+0000 7fcfd4a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:37.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.502+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfd0135a70 con 0x7fcfd013f710 2024-03-25T17:32:37.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.540+0000 7fcfc6ffd640 1 -- 172.21.15.96:0/2614491159 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fcfb80522b0 con 0x7fcfd013f710 2024-03-25T17:32:37.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.540+0000 7fcfc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:37.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.540+0000 7fcfc6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:37.541 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:37.541 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:37.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 10 librados: watch_flush enter 2024-03-25T17:32:37.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 10 librados: watch_flush exit 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 10 client.4383.objecter successfully canceled tick 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 >> v1:172.21.15.203:6800/3611964061 conn(0x7fcfb006dc70 legacy=0x7fcfb0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 10 monclient: shutdown 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 >> v1:172.21.15.96:6789/0 conn(0x7fcfd013f710 legacy=0x7fcfd01db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 shutdown_connections 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 >> 172.21.15.96:0/2614491159 conn(0x7fcfd00b7ea0 msgr2=0x7fcfd014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 shutdown_connections 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 -- 172.21.15.96:0/2614491159 wait complete. 2024-03-25T17:32:37.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:37.542+0000 7fcfd4a0f640 1 librados: shutdown 2024-03-25T17:32:37.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:40.420 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:04.646 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.468 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:04.694 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.503 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:04.729 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.522 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:04.737 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.534 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:04.749 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.546 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:04.772 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.557 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:04.773 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.559 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:04.786 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:40.595 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:04.822 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.363 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:44.479 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.479+0000 7f3c48dcf640 10 monclient: get_monmap_and_config 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.479+0000 7f3c48dcf640 10 monclient: build_initial_monmap 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.479+0000 7f3c48dcf640 10 monclient: monmap: 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:44.480372+0000 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:44.480372+0000 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:44.480 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.480+0000 7f3c48dcf640 1 Processor -- start 2024-03-25T17:32:44.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.480+0000 7f3c48dcf640 1 -- start start 2024-03-25T17:32:44.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.480+0000 7f3c48dcf640 10 monclient: init 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3c440a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3c4413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3c441430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44142520 con 0x7f3c440a98f0 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440b28a0 con 0x7f3c4413f710 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440b2a30 con 0x7f3c441430c0 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:44.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.481+0000 7f3c48dcf640 10 monclient(hunting): authenticate will time out at 5522.763186s 2024-03-25T17:32:44.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.491+0000 7f3c3bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c4413f710 0x7f3c441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:37516/0 (socket says 172.21.15.96:37516) 2024-03-25T17:32:44.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.491+0000 7f3c3bfff640 1 -- 172.21.15.96:0/1788985190 learned_addr learned my addr 172.21.15.96:0/1788985190 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737861437 0 0) 0x7f3c44142520 con 0x7f3c440a98f0 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 10 monclient(hunting): my global_id is 4386 2024-03-25T17:32:44.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.502+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c30003520 con 0x7f3c440a98f0 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446287798 0 0) 0x7f3c30003520 con 0x7f3c440a98f0 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 >> v1:172.21.15.96:6790/0 conn(0x7f3c441430c0 legacy=0x7f3c4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 >> v1:172.21.15.203:6789/0 conn(0x7f3c4413f710 legacy=0x7f3c441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 monclient: found mon.noname-a 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:44.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440b44c0 con 0x7f3c440a98f0 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: _finish_auth 0 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c34001690 con 0x7f3c440a98f0 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c41d9b640 10 monclient: dump: 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:44.514 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c48dcf640 5 monclient: authenticate success, global_id 4386 2024-03-25T17:32:44.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.512+0000 7f3c48dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c340017f0 con 0x7f3c440a98f0 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 1 -- 172.21.15.96:0/1788985190 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c34001c00 con 0x7f3c440a98f0 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c41d9b640 10 monclient: dump: 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:44.531 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c48dcf640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:44.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.530+0000 7f3c48dcf640 4 set_mon_vals no callback set 2024-03-25T17:32:44.533 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 10 monclient: shutdown 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 1 -- 172.21.15.96:0/1788985190 >> v1:172.21.15.96:6789/0 conn(0x7f3c440a98f0 legacy=0x7f3c440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 1 -- 172.21.15.96:0/1788985190 shutdown_connections 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 1 -- 172.21.15.96:0/1788985190 >> 172.21.15.96:0/1788985190 conn(0x7f3c440b7ea0 msgr2=0x7f3c440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 1 -- 172.21.15.96:0/1788985190 shutdown_connections 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.531+0000 7f3c48dcf640 1 -- 172.21.15.96:0/1788985190 wait complete. 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.532+0000 7f3c48dcf640 10 monclient: build_initial_monmap 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.532+0000 7f3c48dcf640 10 monclient: monmap: 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:44.534 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 librados: starting msgr at 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 librados: starting objecter 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 Processor -- start 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 -- start start 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 librados: setting wanted keys 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 1 librados: calling monclient init 2024-03-25T17:32:44.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 10 monclient: init 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.533+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3c4413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3c441430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3c440ae7a0 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44138c80 con 0x7f3c440ae7a0 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440b1920 con 0x7f3c4413f710 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440b05e0 con 0x7f3c441430c0 2024-03-25T17:32:44.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:44.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.534+0000 7f3c48dcf640 10 monclient(hunting): authenticate will time out at 5522.816067s 2024-03-25T17:32:44.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.546+0000 7f3c4259c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c4413f710 0x7f3c440adc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:38938/0 (socket says 172.21.15.96:38938) 2024-03-25T17:32:44.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.546+0000 7f3c4259c640 1 -- 172.21.15.96:0/3393956588 learned_addr learned my addr 172.21.15.96:0/3393956588 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:44.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.547+0000 7f3c3affd640 10 client.?.objecter ms_handle_connect 0x7f3c4413f710 2024-03-25T17:32:44.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.547+0000 7f3c3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:44.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.549+0000 7f3c3affd640 10 client.?.objecter ms_handle_connect 0x7f3c441430c0 2024-03-25T17:32:44.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.549+0000 7f3c3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:44.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.556+0000 7f3c3affd640 10 client.?.objecter ms_handle_connect 0x7f3c440ae7a0 2024-03-25T17:32:44.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.556+0000 7f3c3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:44.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836439825 0 0) 0x7f3c440b05e0 con 0x7f3c441430c0 2024-03-25T17:32:44.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:44.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:44.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:44.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 10 monclient(hunting): my global_id is 4376 2024-03-25T17:32:44.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.559+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c24004eb0 con 0x7f3c441430c0 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409048658 0 0) 0x7f3c44138c80 con 0x7f3c440ae7a0 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 10 monclient(hunting): my global_id is 4389 2024-03-25T17:32:44.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.568+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c440b05e0 con 0x7f3c440ae7a0 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919826162 0 0) 0x7f3c440b1920 con 0x7f3c4413f710 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 10 monclient(hunting): my global_id is 4375 2024-03-25T17:32:44.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.573+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c44138c80 con 0x7f3c4413f710 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108251776 0 0) 0x7f3c440b05e0 con 0x7f3c440ae7a0 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c440b1920 con 0x7f3c440ae7a0 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c3c001b10 con 0x7f3c440ae7a0 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 10 monclient(hunting): dump: 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.579 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.578+0000 7f3c3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2145215172 0 0) 0x7f3c440b1920 con 0x7f3c440ae7a0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 >> v1:172.21.15.96:6790/0 conn(0x7f3c441430c0 legacy=0x7f3c44143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 >> v1:172.21.15.203:6789/0 conn(0x7f3c4413f710 legacy=0x7f3c440adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 1 monclient: found mon.a 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440af2c0 con 0x7f3c440ae7a0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 10 monclient: _finish_auth 0 2024-03-25T17:32:44.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.588+0000 7f3c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 5 monclient: authenticate success, global_id 4389 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 10 monclient: _renew_subs 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c441e34b0 con 0x7f3c440ae7a0 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 10 monclient: _renew_subs 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c441e3820 con 0x7f3c440ae7a0 2024-03-25T17:32:44.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.589+0000 7f3c48dcf640 1 librados: init done 2024-03-25T17:32:44.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.590+0000 7f3c48dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:44.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.590+0000 7f3c48dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:44.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.590+0000 7f3c48dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:44.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.590+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c440b3ef0 con 0x7f3c440ae7a0 2024-03-25T17:32:44.879 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.878+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c3c002000 con 0x7f3c440ae7a0 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.878+0000 7f3c3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.878+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c3c003b30 con 0x7f3c440ae7a0 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.878+0000 7f3c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.879+0000 7f3c3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.879+0000 7f3c3affd640 10 monclient: dump: 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:44.880 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:44.881 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:44.881 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:44.881 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:44.881 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:44.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.878+0000 7f3c4159a640 4 set_mon_vals no callback set 2024-03-25T17:32:44.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.880+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f3c3c003d50 con 0x7f3c440ae7a0 2024-03-25T17:32:44.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.880+0000 7f3c4259c640 10 monclient: get_auth_request con 0x7f3c2406de20 auth_method 0 2024-03-25T17:32:44.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.881+0000 7f3c3affd640 10 client.4389.objecter ms_handle_connect 0x7f3c2406de20 2024-03-25T17:32:44.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f3c3c053830 con 0x7f3c440ae7a0 2024-03-25T17:32:44.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 10 client.4389.objecter ms_dispatch 0x7f3c44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.886+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c3c04f340 con 0x7f3c440ae7a0 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.887+0000 7f3c3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:44.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:44.887+0000 7f3c3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:45.112 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.111+0000 7f3c48dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:45.112 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.112+0000 7f3c48dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:45.112 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.112+0000 7f3c48dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:32:45.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.112+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c440627b0 con 0x7f3c440ae7a0 2024-03-25T17:32:45.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.140+0000 7f3c3affd640 1 -- 172.21.15.96:0/3393956588 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f3c3c051270 con 0x7f3c440ae7a0 2024-03-25T17:32:45.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.140+0000 7f3c3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:45.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.140+0000 7f3c3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:45.141 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:45.141 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 10 librados: watch_flush enter 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 10 librados: watch_flush exit 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 10 client.4389.objecter successfully canceled tick 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 >> v1:172.21.15.203:6800/3611964061 conn(0x7f3c2406de20 legacy=0x7f3c240702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:45.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 10 monclient: shutdown 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 >> v1:172.21.15.96:6789/0 conn(0x7f3c440ae7a0 legacy=0x7f3c440afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 shutdown_connections 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 >> 172.21.15.96:0/3393956588 conn(0x7f3c440b7ea0 msgr2=0x7f3c44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 shutdown_connections 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 -- 172.21.15.96:0/3393956588 wait complete. 2024-03-25T17:32:45.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:45.142+0000 7f3c48dcf640 1 librados: shutdown 2024-03-25T17:32:45.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:51.960 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:52.078 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.078+0000 7f13ccc96640 10 monclient: get_monmap_and_config 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.078+0000 7f13ccc96640 10 monclient: build_initial_monmap 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.078+0000 7f13ccc96640 10 monclient: monmap: 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:52.079220+0000 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:52.079220+0000 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:52.079 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.079+0000 7f13ccc96640 1 Processor -- start 2024-03-25T17:32:52.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.079+0000 7f13ccc96640 1 -- start start 2024-03-25T17:32:52.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.079+0000 7f13ccc96640 10 monclient: init 2024-03-25T17:32:52.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-c con 0x7f13c80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-a con 0x7f13c813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-b con 0x7f13c81430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c8142520 con 0x7f13c813f710 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80b28a0 con 0x7f13c80a98f0 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80b2a30 con 0x7f13c81430c0 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:52.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.080+0000 7f13ccc96640 10 monclient(hunting): authenticate will time out at 5530.361788s 2024-03-25T17:32:52.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.093+0000 7f13c6d9d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f13c81430c0 0x7f13c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:58220/0 (socket says 172.21.15.96:58220) 2024-03-25T17:32:52.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.093+0000 7f13c6d9d640 1 -- 172.21.15.96:0/284102578 learned_addr learned my addr 172.21.15.96:0/284102578 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917219486 0 0) 0x7f13c80b2a30 con 0x7f13c81430c0 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 10 monclient(hunting): _negotiate 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 10 monclient(hunting): my global_id is 4379 2024-03-25T17:32:52.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.101+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b0003520 con 0x7f13c81430c0 2024-03-25T17:32:52.102 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035523195 0 0) 0x7f13c8142520 con 0x7f13c813f710 2024-03-25T17:32:52.102 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 10 monclient(hunting): _negotiate 2024-03-25T17:32:52.102 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:52.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:52.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 10 monclient(hunting): my global_id is 4392 2024-03-25T17:32:52.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.103+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c80b2a30 con 0x7f13c813f710 2024-03-25T17:32:52.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571960123 0 0) 0x7f13b0003520 con 0x7f13c81430c0 2024-03-25T17:32:52.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 >> v1:172.21.15.203:6789/0 conn(0x7f13c80a98f0 legacy=0x7f13c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 >> v1:172.21.15.96:6789/0 conn(0x7f13c813f710 legacy=0x7f13c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: _finish_hunting 0 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 monclient: found mon.noname-b 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c80b44c0 con 0x7f13c81430c0 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: _finish_auth 0 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: _check_auth_tickets 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f13c0001a10 con 0x7f13c81430c0 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13ccc96640 5 monclient: authenticate success, global_id 4379 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13ccc96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:52.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13c559a640 10 monclient: dump: 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.105+0000 7f13ccc96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c0001b70 con 0x7f13c81430c0 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 1 -- 172.21.15.96:0/284102578 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f13c0001f80 con 0x7f13c81430c0 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13ccc96640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13ccc96640 4 set_mon_vals no callback set 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:52.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.126+0000 7f13c559a640 10 monclient: dump: 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:52.127 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 10 monclient: shutdown 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 1 -- 172.21.15.96:0/284102578 >> v1:172.21.15.96:6790/0 conn(0x7f13c81430c0 legacy=0x7f13c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 1 -- 172.21.15.96:0/284102578 shutdown_connections 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 1 -- 172.21.15.96:0/284102578 >> 172.21.15.96:0/284102578 conn(0x7f13c80b7ea0 msgr2=0x7f13c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 1 -- 172.21.15.96:0/284102578 shutdown_connections 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.127+0000 7f13ccc96640 1 -- 172.21.15.96:0/284102578 wait complete. 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.128+0000 7f13ccc96640 10 monclient: build_initial_monmap 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.128+0000 7f13ccc96640 10 monclient: monmap: 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:52.129 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 librados: starting msgr at 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 librados: starting objecter 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 Processor -- start 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 -- start start 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 librados: setting wanted keys 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 1 librados: calling monclient init 2024-03-25T17:32:52.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.129+0000 7f13ccc96640 10 monclient: init 2024-03-25T17:32:52.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:52.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-b con 0x7f13c80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-a con 0x7f13c813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient(hunting): picked mon.noname-c con 0x7f13c81430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c8138b40 con 0x7f13c813f710 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c8138e20 con 0x7f13c80a98f0 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c8139100 con 0x7f13c81430c0 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:52.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.130+0000 7f13ccc96640 10 monclient(hunting): authenticate will time out at 5530.411492s 2024-03-25T17:32:52.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.136+0000 7f13c5d9b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f13c813f710 0x7f13c8137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43512/0 (socket says 172.21.15.96:43512) 2024-03-25T17:32:52.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.136+0000 7f13c5d9b640 1 -- 172.21.15.96:0/2546231984 learned_addr learned my addr 172.21.15.96:0/2546231984 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:52.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.138+0000 7f13b6ffd640 10 client.?.objecter ms_handle_connect 0x7f13c813f710 2024-03-25T17:32:52.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.138+0000 7f13b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:52.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.138+0000 7f13b6ffd640 10 client.?.objecter ms_handle_connect 0x7f13c80a98f0 2024-03-25T17:32:52.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.138+0000 7f13b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:52.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.142+0000 7f13b6ffd640 10 client.?.objecter ms_handle_connect 0x7f13c81430c0 2024-03-25T17:32:52.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.142+0000 7f13b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:52.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444148536 0 0) 0x7f13c8138e20 con 0x7f13c80a98f0 2024-03-25T17:32:52.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:52.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:52.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:52.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T17:32:52.149 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.149+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8004eb0 con 0x7f13c80a98f0 2024-03-25T17:32:52.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 379652184 0 0) 0x7f13a8004eb0 con 0x7f13c80a98f0 2024-03-25T17:32:52.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13c8138e20 con 0x7f13c80a98f0 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f13b80018c0 con 0x7f13c80a98f0 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:52.158 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.157+0000 7f13b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358844777 0 0) 0x7f13c8138b40 con 0x7f13c813f710 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T17:32:52.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.158+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8004eb0 con 0x7f13c813f710 2024-03-25T17:32:52.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360164461 0 0) 0x7f13c8139100 con 0x7f13c81430c0 2024-03-25T17:32:52.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:52.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:52.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:52.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T17:32:52.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.172+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8138b40 con 0x7f13c81430c0 2024-03-25T17:32:52.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4142298863 0 0) 0x7f13c8138e20 con 0x7f13c80a98f0 2024-03-25T17:32:52.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 >> v1:172.21.15.96:6790/0 conn(0x7f13c81430c0 legacy=0x7f13c81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 >> v1:172.21.15.96:6789/0 conn(0x7f13c813f710 legacy=0x7f13c8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 1 monclient: found mon.b 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c81397a0 con 0x7f13c80a98f0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 5 monclient: authenticate success, global_id 4381 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 10 monclient: _renew_subs 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c8136aa0 con 0x7f13c80a98f0 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:52.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 10 monclient: _renew_subs 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c813c320 con 0x7f13c80a98f0 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.173+0000 7f13ccc96640 1 librados: init done 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.174+0000 7f13ccc96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.174+0000 7f13ccc96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:52.175 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.174+0000 7f13ccc96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:52.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.174+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c81395b0 con 0x7f13c80a98f0 2024-03-25T17:32:52.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b80035f0 con 0x7f13c80a98f0 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f13b80039c0 con 0x7f13c80a98f0 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13b6ffd640 10 monclient: dump: 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:52.459 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:52.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.458+0000 7f13c4d99640 4 set_mon_vals no callback set 2024-03-25T17:32:52.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.459+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f13b8003be0 con 0x7f13c80a98f0 2024-03-25T17:32:52.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.460+0000 7f13c5d9b640 10 monclient: get_auth_request con 0x7f13a806de80 auth_method 0 2024-03-25T17:32:52.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.460+0000 7f13b6ffd640 10 client.4381.objecter ms_handle_connect 0x7f13a806de80 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f13b80535d0 con 0x7f13c80a98f0 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 10 client.4381.objecter ms_dispatch 0x7f13c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13b804f0e0 con 0x7f13c80a98f0 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:52.465 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.465+0000 7f13b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:52.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.711+0000 7f13ccc96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:52.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.711+0000 7f13ccc96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:52.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.711+0000 7f13ccc96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:32:52.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.711+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13c8139930 con 0x7f13c80a98f0 2024-03-25T17:32:52.738 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.739+0000 7f13b6ffd640 1 -- 172.21.15.96:0/2546231984 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f13b8051010 con 0x7f13c80a98f0 2024-03-25T17:32:52.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.739+0000 7f13b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:52.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.739+0000 7f13b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:52.739 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:32:52.739 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 10 librados: watch_flush enter 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 10 librados: watch_flush exit 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 10 client.4381.objecter successfully canceled tick 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 >> v1:172.21.15.203:6800/3611964061 conn(0x7f13a806de80 legacy=0x7f13a8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 10 monclient: shutdown 2024-03-25T17:32:52.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 >> v1:172.21.15.203:6789/0 conn(0x7f13c80a98f0 legacy=0x7f13c8137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 shutdown_connections 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 >> 172.21.15.96:0/2546231984 conn(0x7f13c80b7ea0 msgr2=0x7f13c814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 shutdown_connections 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 -- 172.21.15.96:0/2546231984 wait complete. 2024-03-25T17:32:52.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:52.741+0000 7f13ccc96640 1 librados: shutdown 2024-03-25T17:32:52.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:59.560 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.676+0000 7f4d7bea1640 10 monclient: get_monmap_and_config 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.676+0000 7f4d7bea1640 10 monclient: build_initial_monmap 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.676+0000 7f4d7bea1640 10 monclient: monmap: 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:32:59.677737+0000 2024-03-25T17:32:59.677 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:32:59.677737+0000 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:32:59.678 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 1 Processor -- start 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 1 -- start start 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient: init 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-a con 0x7f4d740b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:59.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-c con 0x7f4d740b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-b con 0x7f4d7414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d740af940 con 0x7f4d740b4d60 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d740b0a70 con 0x7f4d740b1730 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d740b0d50 con 0x7f4d7414be90 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:59.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.677+0000 7f4d7bea1640 10 monclient(hunting): authenticate will time out at 5537.959854s 2024-03-25T17:32:59.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.684+0000 7f4d79c3d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4d740b4d60 0x7f4d7413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48016/0 (socket says 172.21.15.96:48016) 2024-03-25T17:32:59.685 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.684+0000 7f4d79c3d640 1 -- 172.21.15.96:0/1299989071 learned_addr learned my addr 172.21.15.96:0/1299989071 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:59.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751927037 0 0) 0x7f4d740af940 con 0x7f4d740b4d60 2024-03-25T17:32:59.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:59.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:59.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:59.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 10 monclient(hunting): my global_id is 4398 2024-03-25T17:32:59.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.702+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d60003520 con 0x7f4d740b4d60 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777676478 0 0) 0x7f4d740b0d50 con 0x7f4d7414be90 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 10 monclient(hunting): my global_id is 4385 2024-03-25T17:32:59.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.711+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d740af940 con 0x7f4d7414be90 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436492339 0 0) 0x7f4d60003520 con 0x7f4d740b4d60 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 >> v1:172.21.15.96:6790/0 conn(0x7f4d7414be90 legacy=0x7f4d7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 >> v1:172.21.15.203:6789/0 conn(0x7f4d740b1730 legacy=0x7f4d740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: _finish_hunting 0 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 monclient: found mon.noname-a 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:32:59.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d740b1030 con 0x7f4d740b4d60 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: _finish_auth 0 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: _check_auth_tickets 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4d70001690 con 0x7f4d740b4d60 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d7bea1640 5 monclient: authenticate success, global_id 4398 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d7bea1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d78c3b640 10 monclient: dump: 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:59.721 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:59.722 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:59.722 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:59.722 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:59.722 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:59.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.719+0000 7f4d7bea1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d700017f0 con 0x7f4d740b4d60 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 1 -- 172.21.15.96:0/1299989071 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4d70001c00 con 0x7f4d740b4d60 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d7bea1640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d7bea1640 4 set_mon_vals no callback set 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.742+0000 7f4d78c3b640 10 monclient: dump: 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.743 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:59.744 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:59.744 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:59.744 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:59.744 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:59.744 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 10 monclient: shutdown 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 1 -- 172.21.15.96:0/1299989071 >> v1:172.21.15.96:6789/0 conn(0x7f4d740b4d60 legacy=0x7f4d7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 1 -- 172.21.15.96:0/1299989071 shutdown_connections 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 1 -- 172.21.15.96:0/1299989071 >> 172.21.15.96:0/1299989071 conn(0x7f4d740617a0 msgr2=0x7f4d74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 1 -- 172.21.15.96:0/1299989071 shutdown_connections 2024-03-25T17:32:59.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 1 -- 172.21.15.96:0/1299989071 wait complete. 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 10 monclient: build_initial_monmap 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.743+0000 7f4d7bea1640 10 monclient: monmap: 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 librados: starting msgr at 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 librados: starting objecter 2024-03-25T17:32:59.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 Processor -- start 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 -- start start 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 librados: setting wanted keys 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 librados: calling monclient init 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient: init 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-a con 0x7f4d740b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-c con 0x7f4d740b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 10 monclient(hunting): picked mon.noname-b con 0x7f4d7414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.744+0000 7f4d7bea1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d741368f0 con 0x7f4d740b1730 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.745+0000 7f4d7bea1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d74137750 con 0x7f4d7414be90 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.745+0000 7f4d7bea1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d741363d0 con 0x7f4d740b4d60 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.745+0000 7f4d7bea1640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:59.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.745+0000 7f4d7bea1640 10 monclient(hunting): authenticate will time out at 5538.026987s 2024-03-25T17:32:59.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.759+0000 7f4d7a43e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f4d7414be90 0x7f4d741d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:39990/0 (socket says 172.21.15.96:39990) 2024-03-25T17:32:59.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.759+0000 7f4d7a43e640 1 -- 172.21.15.96:0/3336630546 learned_addr learned my addr 172.21.15.96:0/3336630546 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:32:59.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.759+0000 7f4d6a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d7414be90 2024-03-25T17:32:59.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.759+0000 7f4d6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:59.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.766+0000 7f4d6a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d740b4d60 2024-03-25T17:32:59.767 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.766+0000 7f4d6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:59.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062104300 0 0) 0x7f4d74137750 con 0x7f4d7414be90 2024-03-25T17:32:59.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:59.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:59.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:59.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T17:32:59.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.771+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d58004eb0 con 0x7f4d7414be90 2024-03-25T17:32:59.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.776+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641207772 0 0) 0x7f4d741363d0 con 0x7f4d740b4d60 2024-03-25T17:32:59.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.777+0000 7f4d6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:59.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.777+0000 7f4d6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:59.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.777+0000 7f4d6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:59.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.777+0000 7f4d6a7fc640 10 monclient(hunting): my global_id is 4388 2024-03-25T17:32:59.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.777+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74137750 con 0x7f4d740b4d60 2024-03-25T17:32:59.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1361662447 0 0) 0x7f4d74137750 con 0x7f4d740b4d60 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d741363d0 con 0x7f4d740b4d60 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4d640018a0 con 0x7f4d740b4d60 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 10 monclient(hunting): dump: 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:32:59.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.788+0000 7f4d6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 840411248 0 0) 0x7f4d741363d0 con 0x7f4d740b4d60 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 >> v1:172.21.15.203:6789/0 conn(0x7f4d7414be90 legacy=0x7f4d741d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 >> v1:172.21.15.96:6789/0 conn(0x7f4d740b1730 legacy=0x7f4d74135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 1 monclient: found mon.c 2024-03-25T17:32:59.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d74130470 con 0x7f4d740b4d60 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 5 monclient: authenticate success, global_id 4388 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 10 monclient: _renew_subs 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d74130720 con 0x7f4d740b4d60 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 10 monclient: _renew_subs 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d741322f0 con 0x7f4d740b4d60 2024-03-25T17:32:59.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.794+0000 7f4d7bea1640 1 librados: init done 2024-03-25T17:32:59.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.795+0000 7f4d7bea1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:59.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.795+0000 7f4d7bea1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:59.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.795+0000 7f4d7bea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:32:59.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:32:59.795+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d74136610 con 0x7f4d740b4d60 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d640035f0 con 0x7f4d740b4d60 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4d64003820 con 0x7f4d740b4d60 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6a7fc640 10 monclient: dump: 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:00.098 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.097+0000 7f4d6bfff640 4 set_mon_vals no callback set 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.098+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f4d64003a40 con 0x7f4d740b4d60 2024-03-25T17:33:00.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.098+0000 7f4d79c3d640 10 monclient: get_auth_request con 0x7f4d5806dd70 auth_method 0 2024-03-25T17:33:00.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.099+0000 7f4d6a7fc640 10 client.4388.objecter ms_handle_connect 0x7f4d5806dd70 2024-03-25T17:33:00.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.103+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f4d64053400 con 0x7f4d740b4d60 2024-03-25T17:33:00.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.103+0000 7f4d6a7fc640 10 client.4388.objecter ms_dispatch 0x7f4d74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:00.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.103+0000 7f4d6a7fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:00.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.103+0000 7f4d6a7fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:00.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.103+0000 7f4d6a7fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T17:33:00.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.104+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d6404ef10 con 0x7f4d740b4d60 2024-03-25T17:33:00.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.104+0000 7f4d6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:00.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.104+0000 7f4d6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:00.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.340+0000 7f4d7bea1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:00.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.340+0000 7f4d7bea1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:00.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.340+0000 7f4d7bea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:00.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.340+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d74136210 con 0x7f4d740b4d60 2024-03-25T17:33:00.390 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.389+0000 7f4d6a7fc640 1 -- 172.21.15.96:0/3336630546 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f4d64050e40 con 0x7f4d740b4d60 2024-03-25T17:33:00.390 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.389+0000 7f4d6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:00.390 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.389+0000 7f4d6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:00.390 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:00.390 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:00.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 10 librados: watch_flush enter 2024-03-25T17:33:00.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 10 librados: watch_flush exit 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 10 client.4388.objecter successfully canceled tick 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 >> v1:172.21.15.203:6800/3611964061 conn(0x7f4d5806dd70 legacy=0x7f4d58070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 10 monclient: shutdown 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 >> v1:172.21.15.96:6790/0 conn(0x7f4d740b4d60 legacy=0x7f4d7412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 shutdown_connections 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 >> 172.21.15.96:0/3336630546 conn(0x7f4d740617a0 msgr2=0x7f4d74138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 shutdown_connections 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.391+0000 7f4d7bea1640 1 -- 172.21.15.96:0/3336630546 wait complete. 2024-03-25T17:33:00.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:00.392+0000 7f4d7bea1640 1 librados: shutdown 2024-03-25T17:33:00.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:07.211 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:07.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.322+0000 7f158a40f640 10 monclient: get_monmap_and_config 2024-03-25T17:33:07.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.322+0000 7f158a40f640 10 monclient: build_initial_monmap 2024-03-25T17:33:07.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.322+0000 7f158a40f640 10 monclient: monmap: 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:07.324049+0000 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:07.324049+0000 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:07.324 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.323+0000 7f158a40f640 1 Processor -- start 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.323+0000 7f158a40f640 1 -- start start 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.323+0000 7f158a40f640 10 monclient: init 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-c con 0x7f15840b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-a con 0x7f15840b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-b con 0x7f158414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15840af940 con 0x7f15840b1730 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15840b0a70 con 0x7f15840b4d60 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15840b0d50 con 0x7f158414be90 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.324+0000 7f158a40f640 10 monclient(hunting): authenticate will time out at 5545.606658s 2024-03-25T17:33:07.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.329+0000 7f15837fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f15840b1730 0x7f15840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58716/0 (socket says 172.21.15.96:58716) 2024-03-25T17:33:07.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.329+0000 7f15837fe640 1 -- 172.21.15.96:0/2192399776 learned_addr learned my addr 172.21.15.96:0/2192399776 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259340927 0 0) 0x7f15840af940 con 0x7f15840b1730 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T17:33:07.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.348+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c003520 con 0x7f15840b1730 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749791163 0 0) 0x7f15840b0a70 con 0x7f15840b4d60 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T17:33:07.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.356+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15840af940 con 0x7f15840b4d60 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243638069 0 0) 0x7f15840b0d50 con 0x7f158414be90 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 10 monclient(hunting): my global_id is 4391 2024-03-25T17:33:07.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.362+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15840b0a70 con 0x7f158414be90 2024-03-25T17:33:07.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074621723 0 0) 0x7f156c003520 con 0x7f15840b1730 2024-03-25T17:33:07.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 >> v1:172.21.15.96:6790/0 conn(0x7f158414be90 legacy=0x7f158414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 >> v1:172.21.15.203:6789/0 conn(0x7f15840b4d60 legacy=0x7f158413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 monclient: found mon.noname-a 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840b1030 con 0x7f15840b1730 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1578001690 con 0x7f15840b1730 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f158a40f640 5 monclient: authenticate success, global_id 4401 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f158a40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f1582ffd640 10 monclient: dump: 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:07.368 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.369 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.366+0000 7f158a40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:07.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15780017f0 con 0x7f15840b1730 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 1 -- 172.21.15.96:0/2192399776 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1578001c00 con 0x7f15840b1730 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 4 set_mon_vals no callback set 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f1582ffd640 10 monclient: dump: 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:07.384 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:07.385 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:07.385 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:07.385 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 10 monclient: shutdown 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 1 -- 172.21.15.96:0/2192399776 >> v1:172.21.15.96:6789/0 conn(0x7f15840b1730 legacy=0x7f15840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 1 -- 172.21.15.96:0/2192399776 shutdown_connections 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 1 -- 172.21.15.96:0/2192399776 >> 172.21.15.96:0/2192399776 conn(0x7f15840617a0 msgr2=0x7f1584061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 1 -- 172.21.15.96:0/2192399776 shutdown_connections 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.383+0000 7f158a40f640 1 -- 172.21.15.96:0/2192399776 wait complete. 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.384+0000 7f158a40f640 10 monclient: build_initial_monmap 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.384+0000 7f158a40f640 10 monclient: monmap: 2024-03-25T17:33:07.387 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.384+0000 7f158a40f640 1 librados: starting msgr at 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.384+0000 7f158a40f640 1 librados: starting objecter 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 Processor -- start 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 -- start start 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 librados: setting wanted keys 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 librados: calling monclient init 2024-03-25T17:33:07.388 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient: init 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-b con 0x7f15840b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-a con 0x7f15840b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient(hunting): picked mon.noname-c con 0x7f158414be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15841451e0 con 0x7f15840b4d60 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15841411f0 con 0x7f15840b1730 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1584141380 con 0x7f158414be90 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:07.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.385+0000 7f158a40f640 10 monclient(hunting): authenticate will time out at 5545.667866s 2024-03-25T17:33:07.390 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.389+0000 7f15889ac640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f158414be90 0x7f1584142ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:44992/0 (socket says 172.21.15.96:44992) 2024-03-25T17:33:07.390 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.389+0000 7f15889ac640 1 -- 172.21.15.96:0/1842503839 learned_addr learned my addr 172.21.15.96:0/1842503839 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:07.391 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.390+0000 7f1580ff9640 10 client.?.objecter ms_handle_connect 0x7f158414be90 2024-03-25T17:33:07.391 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.390+0000 7f1580ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:07.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103043434 0 0) 0x7f1584141380 con 0x7f158414be90 2024-03-25T17:33:07.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:33:07.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:07.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:07.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T17:33:07.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.393+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1570004eb0 con 0x7f158414be90 2024-03-25T17:33:07.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.401+0000 7f1580ff9640 10 client.?.objecter ms_handle_connect 0x7f15840b1730 2024-03-25T17:33:07.402 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.401+0000 7f1580ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:07.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.409+0000 7f1580ff9640 10 client.?.objecter ms_handle_connect 0x7f15840b4d60 2024-03-25T17:33:07.410 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.409+0000 7f1580ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:07.411 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1625819913 0 0) 0x7f1570004eb0 con 0x7f158414be90 2024-03-25T17:33:07.411 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1584141380 con 0x7f158414be90 2024-03-25T17:33:07.411 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f157c001c70 con 0x7f158414be90 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 10 monclient(hunting): dump: 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.412 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.410+0000 7f1580ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:07.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059339086 0 0) 0x7f15841411f0 con 0x7f15840b1730 2024-03-25T17:33:07.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:33:07.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:07.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:07.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 10 monclient(hunting): my global_id is 4393 2024-03-25T17:33:07.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.419+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1570004eb0 con 0x7f15840b1730 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830998624 0 0) 0x7f1584141380 con 0x7f158414be90 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 >> v1:172.21.15.203:6789/0 conn(0x7f15840b1730 legacy=0x7f1584140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 >> v1:172.21.15.96:6789/0 conn(0x7f15840b4d60 legacy=0x7f15841427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 1 monclient: found mon.c 2024-03-25T17:33:07.424 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584137eb0 con 0x7f158414be90 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 10 monclient: _finish_auth 0 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f1580ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 5 monclient: authenticate success, global_id 4394 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 10 monclient: _renew_subs 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1584138100 con 0x7f158414be90 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 10 monclient: _renew_subs 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15841e3710 con 0x7f158414be90 2024-03-25T17:33:07.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.423+0000 7f158a40f640 1 librados: init done 2024-03-25T17:33:07.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.425+0000 7f158a40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:07.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.425+0000 7f158a40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:07.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.425+0000 7f158a40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:07.426 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.425+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15840627b0 con 0x7f158414be90 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f157c003990 con 0x7f158414be90 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f157c003de0 con 0x7f158414be90 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f1580ff9640 10 monclient: dump: 2024-03-25T17:33:07.722 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:07.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.721+0000 7f15827fc640 4 set_mon_vals no callback set 2024-03-25T17:33:07.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.722+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f157c004000 con 0x7f158414be90 2024-03-25T17:33:07.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.723+0000 7f1583fff640 10 monclient: get_auth_request con 0x7f157006dd90 auth_method 0 2024-03-25T17:33:07.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.723+0000 7f1580ff9640 10 client.4394.objecter ms_handle_connect 0x7f157006dd90 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f157c054a00 con 0x7f158414be90 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 10 client.4394.objecter ms_dispatch 0x7f1584000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T17:33:07.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f157c050510 con 0x7f158414be90 2024-03-25T17:33:07.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:07.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.728+0000 7f1580ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:07.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.959+0000 7f158a40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:07.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.959+0000 7f158a40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:07.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.959+0000 7f158a40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:07.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.959+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1584138460 con 0x7f158414be90 2024-03-25T17:33:07.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.990+0000 7f1580ff9640 1 -- 172.21.15.96:0/1842503839 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f157c052440 con 0x7f158414be90 2024-03-25T17:33:07.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.990+0000 7f1580ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:07.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.990+0000 7f1580ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:07.991 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:07.991 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 10 librados: watch_flush enter 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 10 librados: watch_flush exit 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 10 client.4394.objecter successfully canceled tick 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 >> v1:172.21.15.203:6800/3611964061 conn(0x7f157006dd90 legacy=0x7f1570070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 10 monclient: shutdown 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 >> v1:172.21.15.96:6790/0 conn(0x7f158414be90 legacy=0x7f1584142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 shutdown_connections 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 >> 172.21.15.96:0/1842503839 conn(0x7f15840617a0 msgr2=0x7f158413c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 shutdown_connections 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 -- 172.21.15.96:0/1842503839 wait complete. 2024-03-25T17:33:07.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:07.993+0000 7f158a40f640 1 librados: shutdown 2024-03-25T17:33:08.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:14.814 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:14.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 10 monclient: get_monmap_and_config 2024-03-25T17:33:14.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 10 monclient: build_initial_monmap 2024-03-25T17:33:14.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 10 monclient: monmap: 2024-03-25T17:33:14.925 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:14.926307+0000 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:14.926307+0000 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:14.926 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:14.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 1 Processor -- start 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 1 -- start start 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.925+0000 7ff9eabc4640 10 monclient: init 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e4142520 con 0x7ff9e40a98f0 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e40b28a0 con 0x7ff9e41430c0 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e40b2a30 con 0x7ff9e413f710 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:14.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.926+0000 7ff9eabc4640 10 monclient(hunting): authenticate will time out at 5553.208637s 2024-03-25T17:33:14.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.946+0000 7ff9e8960640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff9e40a98f0 0x7ff9e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44074/0 (socket says 172.21.15.96:44074) 2024-03-25T17:33:14.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.946+0000 7ff9e8960640 1 -- 172.21.15.96:0/936303251 learned_addr learned my addr 172.21.15.96:0/936303251 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963725657 0 0) 0x7ff9e4142520 con 0x7ff9e40a98f0 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T17:33:14.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.957+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d4003520 con 0x7ff9e40a98f0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.973+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1237828886 0 0) 0x7ff9d4003520 con 0x7ff9e40a98f0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.973+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 >> v1:172.21.15.96:6790/0 conn(0x7ff9e413f710 legacy=0x7ff9e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 >> v1:172.21.15.203:6789/0 conn(0x7ff9e41430c0 legacy=0x7ff9e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 1 monclient: found mon.noname-a 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40b44c0 con 0x7ff9e40a98f0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: _finish_auth 0 2024-03-25T17:33:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9d8001690 con 0x7ff9e40a98f0 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9e37fe640 10 monclient: dump: 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9eabc4640 5 monclient: authenticate success, global_id 4407 2024-03-25T17:33:14.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.974+0000 7ff9eabc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9d8001860 con 0x7ff9e40a98f0 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 1 -- 172.21.15.96:0/936303251 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9d8001cb0 con 0x7ff9e40a98f0 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9e37fe640 10 monclient: dump: 2024-03-25T17:33:14.991 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9eabc4640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:14.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.990+0000 7ff9eabc4640 4 set_mon_vals no callback set 2024-03-25T17:33:14.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 10 monclient: shutdown 2024-03-25T17:33:14.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 1 -- 172.21.15.96:0/936303251 >> v1:172.21.15.96:6789/0 conn(0x7ff9e40a98f0 legacy=0x7ff9e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 1 -- 172.21.15.96:0/936303251 shutdown_connections 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 1 -- 172.21.15.96:0/936303251 >> 172.21.15.96:0/936303251 conn(0x7ff9e40b7ea0 msgr2=0x7ff9e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 1 -- 172.21.15.96:0/936303251 shutdown_connections 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.991+0000 7ff9eabc4640 1 -- 172.21.15.96:0/936303251 wait complete. 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.992+0000 7ff9eabc4640 10 monclient: build_initial_monmap 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.992+0000 7ff9eabc4640 10 monclient: monmap: 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:14.994 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 librados: starting msgr at 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 librados: starting objecter 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 Processor -- start 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 -- start start 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 librados: setting wanted keys 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 1 librados: calling monclient init 2024-03-25T17:33:14.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 10 monclient: init 2024-03-25T17:33:14.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:14.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:33:14.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:14.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.993+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e41d76e0 con 0x7ff9e413f710 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e40abd80 con 0x7ff9e41430c0 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e4137c90 con 0x7ff9e40a98f0 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:14.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:14.994+0000 7ff9eabc4640 10 monclient(hunting): authenticate will time out at 5553.276103s 2024-03-25T17:33:15.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.006+0000 7ff9e8960640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ff9e40a98f0 0x7ff9e41d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:39794/0 (socket says 172.21.15.96:39794) 2024-03-25T17:33:15.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.006+0000 7ff9e8960640 1 -- 172.21.15.96:0/2384737934 learned_addr learned my addr 172.21.15.96:0/2384737934 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.011+0000 7ff9e17fa640 10 client.?.objecter ms_handle_connect 0x7ff9e40a98f0 2024-03-25T17:33:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.011+0000 7ff9e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:15.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.018+0000 7ff9e17fa640 10 client.?.objecter ms_handle_connect 0x7ff9e413f710 2024-03-25T17:33:15.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.018+0000 7ff9e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:15.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.022+0000 7ff9e17fa640 10 client.?.objecter ms_handle_connect 0x7ff9e41430c0 2024-03-25T17:33:15.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.022+0000 7ff9e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:15.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900885155 0 0) 0x7ff9e4137c90 con 0x7ff9e40a98f0 2024-03-25T17:33:15.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:33:15.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:15.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:15.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 10 monclient(hunting): my global_id is 4397 2024-03-25T17:33:15.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.026+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c8004eb0 con 0x7ff9e40a98f0 2024-03-25T17:33:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727748601 0 0) 0x7ff9e41d76e0 con 0x7ff9e413f710 2024-03-25T17:33:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): my global_id is 4410 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e4137c90 con 0x7ff9e413f710 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218648529 0 0) 0x7ff9e40abd80 con 0x7ff9e41430c0 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 10 monclient(hunting): my global_id is 4396 2024-03-25T17:33:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.033+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e41d76e0 con 0x7ff9e41430c0 2024-03-25T17:33:15.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875980886 0 0) 0x7ff9c8004eb0 con 0x7ff9e40a98f0 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e40abd80 con 0x7ff9e40a98f0 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9d8001b70 con 0x7ff9e40a98f0 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): dump: 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:15.048 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:15.049 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:15.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:15.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123228574 0 0) 0x7ff9e4137c90 con 0x7ff9e413f710 2024-03-25T17:33:15.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9c8004eb0 con 0x7ff9e413f710 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9d0001810 con 0x7ff9e413f710 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.047+0000 7ff9e17fa640 10 monclient(hunting): dump: 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:15.050 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143859824 0 0) 0x7ff9e41d76e0 con 0x7ff9e41430c0 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e4137c90 con 0x7ff9e41430c0 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9dc001a10 con 0x7ff9e41430c0 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.048+0000 7ff9e17fa640 10 monclient(hunting): dump: 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:15.051 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:15.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3014483130 0 0) 0x7ff9e4137c90 con 0x7ff9e41430c0 2024-03-25T17:33:15.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 >> v1:172.21.15.96:6790/0 conn(0x7ff9e40a98f0 legacy=0x7ff9e41d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:15.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 >> v1:172.21.15.96:6789/0 conn(0x7ff9e413f710 legacy=0x7ff9e4138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:15.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:33:15.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 1 monclient: found mon.b 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e41d41e0 con 0x7ff9e41430c0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 10 monclient: _finish_auth 0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 5 monclient: authenticate success, global_id 4396 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 10 monclient: _renew_subs 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e41d82d0 con 0x7ff9e41430c0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 10 monclient: _renew_subs 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e4138d80 con 0x7ff9e41430c0 2024-03-25T17:33:15.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.053+0000 7ff9eabc4640 1 librados: init done 2024-03-25T17:33:15.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.054+0000 7ff9eabc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:15.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.054+0000 7ff9eabc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:15.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.054+0000 7ff9eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:15.056 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.054+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e40627b0 con 0x7ff9e41430c0 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9dc001f20 con 0x7ff9e41430c0 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff9dc003b10 con 0x7ff9e41430c0 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e17fa640 10 monclient: dump: 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:15.341 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:15.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.340+0000 7ff9e2ffd640 4 set_mon_vals no callback set 2024-03-25T17:33:15.343 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.341+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ff9dc019ae0 con 0x7ff9e41430c0 2024-03-25T17:33:15.343 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.342+0000 7ff9e8960640 10 monclient: get_auth_request con 0x7ff9c806e0c0 auth_method 0 2024-03-25T17:33:15.343 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.342+0000 7ff9e17fa640 10 client.4396.objecter ms_handle_connect 0x7ff9c806e0c0 2024-03-25T17:33:15.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.350+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ff9dc053770 con 0x7ff9e41430c0 2024-03-25T17:33:15.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.350+0000 7ff9e17fa640 10 client.4396.objecter ms_dispatch 0x7ff9e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9dc04f280 con 0x7ff9e41430c0 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:15.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.351+0000 7ff9e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:15.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.592+0000 7ff9eabc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:15.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.593+0000 7ff9eabc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:15.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.593+0000 7ff9eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:15.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.593+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9e413b430 con 0x7ff9e41430c0 2024-03-25T17:33:15.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.617+0000 7ff9e17fa640 1 -- 172.21.15.96:0/2384737934 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ff9dc0511b0 con 0x7ff9e41430c0 2024-03-25T17:33:15.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.617+0000 7ff9e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:15.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.617+0000 7ff9e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:15.618 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:15.618 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.619+0000 7ff9eabc4640 10 librados: watch_flush enter 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.619+0000 7ff9eabc4640 10 librados: watch_flush exit 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.619+0000 7ff9eabc4640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.619+0000 7ff9eabc4640 10 client.4396.objecter successfully canceled tick 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 >> v1:172.21.15.203:6800/3611964061 conn(0x7ff9c806e0c0 legacy=0x7ff9c8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 10 monclient: shutdown 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 >> v1:172.21.15.203:6789/0 conn(0x7ff9e41430c0 legacy=0x7ff9e41e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 shutdown_connections 2024-03-25T17:33:15.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 >> 172.21.15.96:0/2384737934 conn(0x7ff9e40b7ea0 msgr2=0x7ff9e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:15.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 shutdown_connections 2024-03-25T17:33:15.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 -- 172.21.15.96:0/2384737934 wait complete. 2024-03-25T17:33:15.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:15.620+0000 7ff9eabc4640 1 librados: shutdown 2024-03-25T17:33:15.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:20.598 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:44.824 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.609 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:44.835 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.654 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:04:44.880 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.663 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:44.878 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.672 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:44.887 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.679 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:44.906 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.686 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:44.913 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.701 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:44.916 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:20.725 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:04:44.952 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:22.438 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.554+0000 7f6e695e6640 10 monclient: get_monmap_and_config 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.554+0000 7f6e695e6640 10 monclient: build_initial_monmap 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.554+0000 7f6e695e6640 10 monclient: monmap: 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:22.555499+0000 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:22.555499+0000 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:22.555 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.556 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:22.556 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:22.556 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:22.556 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 1 Processor -- start 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 1 -- start start 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient: init 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e640a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:22.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e641430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:22.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.555+0000 7f6e695e6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e64142520 con 0x7f6e641430c0 2024-03-25T17:33:22.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.556+0000 7f6e695e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640b28a0 con 0x7f6e6413f710 2024-03-25T17:33:22.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.556+0000 7f6e695e6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640b2a30 con 0x7f6e640a98f0 2024-03-25T17:33:22.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.556+0000 7f6e695e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:22.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.556+0000 7f6e695e6640 10 monclient(hunting): authenticate will time out at 5560.837991s 2024-03-25T17:33:22.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.574+0000 7f6e627fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e6413f710 0x7f6e641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33918/0 (socket says 172.21.15.96:33918) 2024-03-25T17:33:22.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.574+0000 7f6e627fc640 1 -- 172.21.15.96:0/514097609 learned_addr learned my addr 172.21.15.96:0/514097609 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933832427 0 0) 0x7f6e640b2a30 con 0x7f6e640a98f0 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T17:33:22.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.587+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c003520 con 0x7f6e640a98f0 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253884107 0 0) 0x7f6e64142520 con 0x7f6e641430c0 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 10 monclient(hunting): my global_id is 4413 2024-03-25T17:33:22.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.593+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e640b2a30 con 0x7f6e641430c0 2024-03-25T17:33:22.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471250491 0 0) 0x7f6e640b28a0 con 0x7f6e6413f710 2024-03-25T17:33:22.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 10 monclient(hunting): my global_id is 4399 2024-03-25T17:33:22.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.595+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e64142520 con 0x7f6e6413f710 2024-03-25T17:33:22.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659787242 0 0) 0x7f6e64142520 con 0x7f6e6413f710 2024-03-25T17:33:22.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 >> v1:172.21.15.96:6790/0 conn(0x7f6e640a98f0 legacy=0x7f6e640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:22.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 >> v1:172.21.15.96:6789/0 conn(0x7f6e641430c0 legacy=0x7f6e6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:22.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:33:22.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 monclient: found mon.noname-c 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640b44c0 con 0x7f6e6413f710 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: _finish_auth 0 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e580015b0 con 0x7f6e6413f710 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e695e6640 5 monclient: authenticate success, global_id 4399 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e695e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e61ffb640 10 monclient: dump: 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.611 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.609+0000 7f6e695e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e58001710 con 0x7f6e6413f710 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 1 -- 172.21.15.96:0/514097609 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e58001b60 con 0x7f6e6413f710 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e695e6640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e695e6640 4 set_mon_vals no callback set 2024-03-25T17:33:22.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.632+0000 7f6e61ffb640 10 monclient: dump: 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.634 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 10 monclient: shutdown 2024-03-25T17:33:22.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 1 -- 172.21.15.96:0/514097609 >> v1:172.21.15.203:6789/0 conn(0x7f6e6413f710 legacy=0x7f6e641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 1 -- 172.21.15.96:0/514097609 shutdown_connections 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 1 -- 172.21.15.96:0/514097609 >> 172.21.15.96:0/514097609 conn(0x7f6e640b7ea0 msgr2=0x7f6e640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.633+0000 7f6e695e6640 1 -- 172.21.15.96:0/514097609 shutdown_connections 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.634+0000 7f6e695e6640 1 -- 172.21.15.96:0/514097609 wait complete. 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.634+0000 7f6e695e6640 10 monclient: build_initial_monmap 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.634+0000 7f6e695e6640 10 monclient: monmap: 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:22.636 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:22.637 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:22.637 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 librados: starting msgr at 2024-03-25T17:33:22.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 librados: starting objecter 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 Processor -- start 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 -- start start 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 librados: setting wanted keys 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 1 librados: calling monclient init 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.635+0000 7f6e695e6640 10 monclient: init 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e640a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e641430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:22.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640abb90 con 0x7f6e640a98f0 2024-03-25T17:33:22.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e641da920 con 0x7f6e641430c0 2024-03-25T17:33:22.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e64137790 con 0x7f6e6413f710 2024-03-25T17:33:22.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:22.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.636+0000 7f6e695e6640 10 monclient(hunting): authenticate will time out at 5560.918552s 2024-03-25T17:33:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.650+0000 7f6e637fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e641430c0 0x7f6e641e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33966/0 (socket says 172.21.15.96:33966) 2024-03-25T17:33:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.650+0000 7f6e637fe640 1 -- 172.21.15.96:0/2107019610 learned_addr learned my addr 172.21.15.96:0/2107019610 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.653+0000 7f6e4b7fe640 10 client.?.objecter ms_handle_connect 0x7f6e6413f710 2024-03-25T17:33:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.653+0000 7f6e4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:22.655 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.654+0000 7f6e4b7fe640 10 client.?.objecter ms_handle_connect 0x7f6e641430c0 2024-03-25T17:33:22.655 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.654+0000 7f6e4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:22.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.657+0000 7f6e4b7fe640 10 client.?.objecter ms_handle_connect 0x7f6e640a98f0 2024-03-25T17:33:22.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.657+0000 7f6e4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810671265 0 0) 0x7f6e641da920 con 0x7f6e641430c0 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T17:33:22.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.660+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e50004eb0 con 0x7f6e641430c0 2024-03-25T17:33:22.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118740001 0 0) 0x7f6e64137790 con 0x7f6e6413f710 2024-03-25T17:33:22.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 10 monclient(hunting): my global_id is 4403 2024-03-25T17:33:22.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.665+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e641da920 con 0x7f6e6413f710 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016912036 0 0) 0x7f6e640abb90 con 0x7f6e640a98f0 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 10 monclient(hunting): my global_id is 4416 2024-03-25T17:33:22.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.669+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e64137790 con 0x7f6e640a98f0 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 669979756 0 0) 0x7f6e50004eb0 con 0x7f6e641430c0 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e640abb90 con 0x7f6e641430c0 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e5c001c90 con 0x7f6e641430c0 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.675 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.674+0000 7f6e4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:22.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 384356753 0 0) 0x7f6e641da920 con 0x7f6e6413f710 2024-03-25T17:33:22.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e50004eb0 con 0x7f6e6413f710 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e58001a20 con 0x7f6e6413f710 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.681+0000 7f6e4b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.682 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.683 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157881220 0 0) 0x7f6e64137790 con 0x7f6e640a98f0 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e641da920 con 0x7f6e640a98f0 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e54001950 con 0x7f6e640a98f0 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.685+0000 7f6e4b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.686 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.687 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1259600423 0 0) 0x7f6e640abb90 con 0x7f6e641430c0 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 >> v1:172.21.15.96:6790/0 conn(0x7f6e6413f710 legacy=0x7f6e64138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 >> v1:172.21.15.96:6789/0 conn(0x7f6e640a98f0 legacy=0x7f6e641dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 1 monclient: found mon.b 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64137920 con 0x7f6e641430c0 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:33:22.693 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 5 monclient: authenticate success, global_id 4402 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 10 monclient: _renew_subs 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641db550 con 0x7f6e641430c0 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 10 monclient: _renew_subs 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e64138c00 con 0x7f6e641430c0 2024-03-25T17:33:22.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.692+0000 7f6e695e6640 1 librados: init done 2024-03-25T17:33:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.693+0000 7f6e695e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.693+0000 7f6e695e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.693+0000 7f6e695e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.693+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e641dab60 con 0x7f6e641430c0 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.970+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e5c003990 con 0x7f6e641430c0 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.970+0000 7f6e4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.970+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e5c003b80 con 0x7f6e641430c0 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.971+0000 7f6e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.971+0000 7f6e4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.971+0000 7f6e4b7fe640 10 monclient: dump: 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.972 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:22.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.971+0000 7f6e617fa640 4 set_mon_vals no callback set 2024-03-25T17:33:22.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.972+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f6e5c003da0 con 0x7f6e641430c0 2024-03-25T17:33:22.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.972+0000 7f6e62ffd640 10 monclient: get_auth_request con 0x7f6e5006e0f0 auth_method 0 2024-03-25T17:33:22.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.973+0000 7f6e4b7fe640 10 client.4402.objecter ms_handle_connect 0x7f6e5006e0f0 2024-03-25T17:33:22.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.975+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f6e5c054970 con 0x7f6e641430c0 2024-03-25T17:33:22.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.975+0000 7f6e4b7fe640 10 client.4402.objecter ms_dispatch 0x7f6e64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.975+0000 7f6e4b7fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.975+0000 7f6e4b7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.976+0000 7f6e4b7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.976+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e5c050480 con 0x7f6e641430c0 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.976+0000 7f6e4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:22.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:22.976+0000 7f6e4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:23.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.199+0000 7f6e695e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:23.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.199+0000 7f6e695e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:23.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.199+0000 7f6e695e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:23.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.199+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e640627b0 con 0x7f6e641430c0 2024-03-25T17:33:23.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.376+0000 7f6e4b7fe640 1 -- 172.21.15.96:0/2107019610 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f6e5c0523b0 con 0x7f6e641430c0 2024-03-25T17:33:23.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.376+0000 7f6e4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:23.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.376+0000 7f6e4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:23.378 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:23.378 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 10 librados: watch_flush enter 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 10 librados: watch_flush exit 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 10 client.4402.objecter successfully canceled tick 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 >> v1:172.21.15.203:6800/3611964061 conn(0x7f6e5006e0f0 legacy=0x7f6e500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 10 monclient: shutdown 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 >> v1:172.21.15.203:6789/0 conn(0x7f6e641430c0 legacy=0x7f6e641e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:23.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 shutdown_connections 2024-03-25T17:33:23.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.378+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 >> 172.21.15.96:0/2107019610 conn(0x7f6e640b7ea0 msgr2=0x7f6e64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:23.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.379+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 shutdown_connections 2024-03-25T17:33:23.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.379+0000 7f6e695e6640 1 -- 172.21.15.96:0/2107019610 wait complete. 2024-03-25T17:33:23.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:23.379+0000 7f6e695e6640 1 librados: shutdown 2024-03-25T17:33:23.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:30.198 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.316+0000 7f56be4c5640 10 monclient: get_monmap_and_config 2024-03-25T17:33:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.316+0000 7f56be4c5640 10 monclient: build_initial_monmap 2024-03-25T17:33:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.316+0000 7f56be4c5640 10 monclient: monmap: 2024-03-25T17:33:30.316 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:30.316 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:30.316977+0000 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:30.316977+0000 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:30.317 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.317+0000 7f56be4c5640 1 Processor -- start 2024-03-25T17:33:30.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.317+0000 7f56be4c5640 1 -- start start 2024-03-25T17:33:30.318 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.317+0000 7f56be4c5640 10 monclient: init 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f56b80a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f56b813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f56b81430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b8142520 con 0x7f56b80a98f0 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b80b28a0 con 0x7f56b813f710 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b80b2a30 con 0x7f56b81430c0 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:30.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.318+0000 7f56be4c5640 10 monclient(hunting): authenticate will time out at 5568.599595s 2024-03-25T17:33:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.333+0000 7f56b77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f56b813f710 0x7f56b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:55932/0 (socket says 172.21.15.96:55932) 2024-03-25T17:33:30.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.333+0000 7f56b77fe640 1 -- 172.21.15.96:0/65700398 learned_addr learned my addr 172.21.15.96:0/65700398 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919809080 0 0) 0x7f56b80b28a0 con 0x7f56b813f710 2024-03-25T17:33:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4405 2024-03-25T17:33:30.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.350+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a0003520 con 0x7f56b813f710 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921639197 0 0) 0x7f56b80b2a30 con 0x7f56b81430c0 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T17:33:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.357+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b80b28a0 con 0x7f56b81430c0 2024-03-25T17:33:30.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286094280 0 0) 0x7f56b8142520 con 0x7f56b80a98f0 2024-03-25T17:33:30.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 10 monclient(hunting): my global_id is 4419 2024-03-25T17:33:30.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.371+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b80b2a30 con 0x7f56b80a98f0 2024-03-25T17:33:30.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1247651665 0 0) 0x7f56a0003520 con 0x7f56b813f710 2024-03-25T17:33:30.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 >> v1:172.21.15.96:6790/0 conn(0x7f56b81430c0 legacy=0x7f56b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 >> v1:172.21.15.96:6789/0 conn(0x7f56b80a98f0 legacy=0x7f56b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 monclient: found mon.noname-c 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80b44c0 con 0x7f56b813f710 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f56ac001690 con 0x7f56b813f710 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56b6ffd640 10 monclient: dump: 2024-03-25T17:33:30.378 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56be4c5640 5 monclient: authenticate success, global_id 4405 2024-03-25T17:33:30.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.377+0000 7f56be4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56ac0017f0 con 0x7f56b813f710 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 1 -- 172.21.15.96:0/65700398 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f56ac001c00 con 0x7f56b813f710 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56be4c5640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56be4c5640 4 set_mon_vals no callback set 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:30.389 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.389+0000 7f56b6ffd640 10 monclient: dump: 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:30.390 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 10 monclient: shutdown 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 1 -- 172.21.15.96:0/65700398 >> v1:172.21.15.203:6789/0 conn(0x7f56b813f710 legacy=0x7f56b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 1 -- 172.21.15.96:0/65700398 shutdown_connections 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 1 -- 172.21.15.96:0/65700398 >> 172.21.15.96:0/65700398 conn(0x7f56b80b7ea0 msgr2=0x7f56b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 1 -- 172.21.15.96:0/65700398 shutdown_connections 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.390+0000 7f56be4c5640 1 -- 172.21.15.96:0/65700398 wait complete. 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.391+0000 7f56be4c5640 10 monclient: build_initial_monmap 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.391+0000 7f56be4c5640 10 monclient: monmap: 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:30.392 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 librados: starting msgr at 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 librados: starting objecter 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 Processor -- start 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 -- start start 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 librados: setting wanted keys 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 1 librados: calling monclient init 2024-03-25T17:33:30.393 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 10 monclient: init 2024-03-25T17:33:30.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:30.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:33:30.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f56b80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.392+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f56b813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f56b81430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b8138a50 con 0x7f56b813f710 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b80ab9e0 con 0x7f56b80a98f0 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b81d7880 con 0x7f56b81430c0 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:30.395 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.393+0000 7f56be4c5640 10 monclient(hunting): authenticate will time out at 5568.674091s 2024-03-25T17:33:30.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.397+0000 7f56b7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f56b80a98f0 0x7f56b8138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:55984/0 (socket says 172.21.15.96:55984) 2024-03-25T17:33:30.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.397+0000 7f56b7fff640 1 -- 172.21.15.96:0/2161598570 learned_addr learned my addr 172.21.15.96:0/2161598570 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:30.400 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.400+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b80a98f0 2024-03-25T17:33:30.400 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.400+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:30.413 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424801471 0 0) 0x7f56b80ab9e0 con 0x7f56b80a98f0 2024-03-25T17:33:30.413 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T17:33:30.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.414+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4004eb0 con 0x7f56b80a98f0 2024-03-25T17:33:30.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.418+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b81430c0 2024-03-25T17:33:30.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.418+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:30.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.418+0000 7f56b4ff9640 10 client.?.objecter ms_handle_connect 0x7f56b813f710 2024-03-25T17:33:30.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.418+0000 7f56b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052542308 0 0) 0x7f56b8138a50 con 0x7f56b813f710 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4422 2024-03-25T17:33:30.428 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.428+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b80ab9e0 con 0x7f56b813f710 2024-03-25T17:33:30.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163893180 0 0) 0x7f56b81d7880 con 0x7f56b81430c0 2024-03-25T17:33:30.430 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:33:30.430 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:30.430 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:30.430 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 10 monclient(hunting): my global_id is 4409 2024-03-25T17:33:30.430 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.430+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b8138a50 con 0x7f56b81430c0 2024-03-25T17:33:30.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437557924 0 0) 0x7f56a4004eb0 con 0x7f56b80a98f0 2024-03-25T17:33:30.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56b81d7880 con 0x7f56b80a98f0 2024-03-25T17:33:30.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f56a8001a10 con 0x7f56b80a98f0 2024-03-25T17:33:30.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:30.431 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 10 monclient(hunting): dump: 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.432 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.431+0000 7f56b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 297770109 0 0) 0x7f56b81d7880 con 0x7f56b80a98f0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 >> v1:172.21.15.96:6790/0 conn(0x7f56b81430c0 legacy=0x7f56b81e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 >> v1:172.21.15.96:6789/0 conn(0x7f56b813f710 legacy=0x7f56b81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 1 monclient: found mon.b 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b81d84b0 con 0x7f56b80a98f0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 10 monclient: _finish_auth 0 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 5 monclient: authenticate success, global_id 4408 2024-03-25T17:33:30.445 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 10 monclient: _renew_subs 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b81d51f0 con 0x7f56b80a98f0 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 10 monclient: _renew_subs 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8138e60 con 0x7f56b80a98f0 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.445+0000 7f56be4c5640 1 librados: init done 2024-03-25T17:33:30.446 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.446+0000 7f56be4c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:30.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.446+0000 7f56be4c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:30.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.446+0000 7f56be4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:30.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.446+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b81d76e0 con 0x7f56b80a98f0 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a8001c70 con 0x7f56b80a98f0 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f56a80038c0 con 0x7f56b80a98f0 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b4ff9640 10 monclient: dump: 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:30.769 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:30.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.769+0000 7f56b67fc640 4 set_mon_vals no callback set 2024-03-25T17:33:30.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.770+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f56a8003aa0 con 0x7f56b80a98f0 2024-03-25T17:33:30.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.771+0000 7f56b77fe640 10 monclient: get_auth_request con 0x7f56a406ddf0 auth_method 0 2024-03-25T17:33:30.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.771+0000 7f56b4ff9640 10 client.4408.objecter ms_handle_connect 0x7f56a406ddf0 2024-03-25T17:33:30.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.772+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f56a8054750 con 0x7f56b80a98f0 2024-03-25T17:33:30.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.772+0000 7f56b4ff9640 10 client.4408.objecter ms_dispatch 0x7f56b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:30.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.772+0000 7f56b4ff9640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:30.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.772+0000 7f56b4ff9640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:30.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.772+0000 7f56b4ff9640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T17:33:30.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.776+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56a8050090 con 0x7f56b80a98f0 2024-03-25T17:33:30.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.776+0000 7f56b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:30.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:30.776+0000 7f56b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:31.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.008+0000 7f56be4c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:31.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.008+0000 7f56be4c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:31.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.008+0000 7f56be4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:31.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.008+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56b81d7a10 con 0x7f56b80a98f0 2024-03-25T17:33:31.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.159+0000 7f56b4ff9640 1 -- 172.21.15.96:0/2161598570 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f56a8052190 con 0x7f56b80a98f0 2024-03-25T17:33:31.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.159+0000 7f56b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:31.159 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.159+0000 7f56b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:31.160 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:31.160 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:31.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 10 librados: watch_flush enter 2024-03-25T17:33:31.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 10 librados: watch_flush exit 2024-03-25T17:33:31.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T17:33:31.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 10 client.4408.objecter successfully canceled tick 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 >> v1:172.21.15.203:6800/3611964061 conn(0x7f56a406ddf0 legacy=0x7f56a40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 10 monclient: shutdown 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 >> v1:172.21.15.203:6789/0 conn(0x7f56b80a98f0 legacy=0x7f56b8138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 shutdown_connections 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 >> 172.21.15.96:0/2161598570 conn(0x7f56b80b7ea0 msgr2=0x7f56b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 shutdown_connections 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 -- 172.21.15.96:0/2161598570 wait complete. 2024-03-25T17:33:31.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:31.161+0000 7f56be4c5640 1 librados: shutdown 2024-03-25T17:33:31.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:37.980 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:38.097 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.098+0000 7f62c46c8640 10 monclient: get_monmap_and_config 2024-03-25T17:33:38.097 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.098+0000 7f62c46c8640 10 monclient: build_initial_monmap 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.098+0000 7f62c46c8640 10 monclient: monmap: 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:38.098364+0000 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:38.098364+0000 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:38.098 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.098+0000 7f62c46c8640 1 Processor -- start 2024-03-25T17:33:38.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 1 -- start start 2024-03-25T17:33:38.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 10 monclient: init 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.099+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc13f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc1430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc142520 con 0x7f62bc13f710 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b28a0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b2a30 con 0x7f62bc1430c0 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:38.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.100+0000 7f62c46c8640 10 monclient(hunting): authenticate will time out at 5576.381004s 2024-03-25T17:33:38.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.118+0000 7f62c1c63640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f62bc13f710 0x7f62bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38320/0 (socket says 172.21.15.96:38320) 2024-03-25T17:33:38.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.118+0000 7f62c1c63640 1 -- 172.21.15.96:0/2990869808 learned_addr learned my addr 172.21.15.96:0/2990869808 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:38.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994014082 0 0) 0x7f62bc142520 con 0x7f62bc13f710 2024-03-25T17:33:38.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 10 monclient(hunting): _negotiate 2024-03-25T17:33:38.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:38.127 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:38.127 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 10 monclient(hunting): my global_id is 4425 2024-03-25T17:33:38.127 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.127+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8003520 con 0x7f62bc13f710 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700688639 0 0) 0x7f62a8003520 con 0x7f62bc13f710 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 >> v1:172.21.15.96:6790/0 conn(0x7f62bc1430c0 legacy=0x7f62bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 >> v1:172.21.15.203:6789/0 conn(0x7f62bc0a98f0 legacy=0x7f62bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: _finish_hunting 0 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 monclient: found mon.noname-a 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0b44c0 con 0x7f62bc13f710 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: _finish_auth 0 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: _check_auth_tickets 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f62ac001690 con 0x7f62bc13f710 2024-03-25T17:33:38.133 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c46c8640 5 monclient: authenticate success, global_id 4425 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c46c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c1462640 10 monclient: dump: 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.133+0000 7f62c46c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:38.147 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62ac0017f0 con 0x7f62bc13f710 2024-03-25T17:33:38.147 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:38.147 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 1 -- 172.21.15.96:0/2990869808 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f62ac001c00 con 0x7f62bc13f710 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c1462640 10 monclient: dump: 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c46c8640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:38.148 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c46c8640 4 set_mon_vals no callback set 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c46c8640 10 monclient: shutdown 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c46c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.148+0000 7f62c46c8640 1 -- 172.21.15.96:0/2990869808 >> v1:172.21.15.96:6789/0 conn(0x7f62bc13f710 legacy=0x7f62bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.149+0000 7f62c46c8640 1 -- 172.21.15.96:0/2990869808 shutdown_connections 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.149+0000 7f62c46c8640 1 -- 172.21.15.96:0/2990869808 >> 172.21.15.96:0/2990869808 conn(0x7f62bc0b7ea0 msgr2=0x7f62bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.149+0000 7f62c46c8640 1 -- 172.21.15.96:0/2990869808 shutdown_connections 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.149+0000 7f62c46c8640 1 -- 172.21.15.96:0/2990869808 wait complete. 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 10 monclient: build_initial_monmap 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 10 monclient: monmap: 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:38.150 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 librados: starting msgr at 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 librados: starting objecter 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 Processor -- start 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 -- start start 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 librados: setting wanted keys 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 1 librados: calling monclient init 2024-03-25T17:33:38.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.150+0000 7f62c46c8640 10 monclient: init 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc1430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0abd70 con 0x7f62bc1430c0 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc1d7710 con 0x7f62bc13f710 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc1d8340 con 0x7f62bc0a98f0 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:38.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.151+0000 7f62c46c8640 10 monclient(hunting): authenticate will time out at 5576.432232s 2024-03-25T17:33:38.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.157+0000 7f62c2464640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f62bc0a98f0 0x7f62bc137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:43722/0 (socket says 172.21.15.96:43722) 2024-03-25T17:33:38.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.157+0000 7f62c2464640 1 -- 172.21.15.96:0/1396768897 learned_addr learned my addr 172.21.15.96:0/1396768897 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:38.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.158+0000 7f62b2ffd640 10 client.?.objecter ms_handle_connect 0x7f62bc0a98f0 2024-03-25T17:33:38.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.158+0000 7f62b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:38.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.162+0000 7f62b2ffd640 10 client.?.objecter ms_handle_connect 0x7f62bc13f710 2024-03-25T17:33:38.162 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.162+0000 7f62b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:38.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500521185 0 0) 0x7f62bc1d8340 con 0x7f62bc0a98f0 2024-03-25T17:33:38.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:38.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:38.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:38.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 10 monclient(hunting): my global_id is 4415 2024-03-25T17:33:38.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.170+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a0004eb0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.174+0000 7f62b2ffd640 10 client.?.objecter ms_handle_connect 0x7f62bc1430c0 2024-03-25T17:33:38.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.174+0000 7f62b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360486570 0 0) 0x7f62bc1d7710 con 0x7f62bc13f710 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 10 monclient(hunting): my global_id is 4414 2024-03-25T17:33:38.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.176+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc1d8340 con 0x7f62bc13f710 2024-03-25T17:33:38.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075011997 0 0) 0x7f62a0004eb0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62bc1d7710 con 0x7f62bc0a98f0 2024-03-25T17:33:38.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f62b8001a30 con 0x7f62bc0a98f0 2024-03-25T17:33:38.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:38.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 10 monclient(hunting): dump: 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.179+0000 7f62b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431159828 0 0) 0x7f62bc1d8340 con 0x7f62bc13f710 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62a0004eb0 con 0x7f62bc13f710 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f62ac001b40 con 0x7f62bc13f710 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:38.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.188+0000 7f62b2ffd640 10 monclient(hunting): dump: 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:38.189 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2865003659 0 0) 0x7f62bc1d7710 con 0x7f62bc0a98f0 2024-03-25T17:33:38.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 >> v1:172.21.15.203:6789/0 conn(0x7f62bc13f710 legacy=0x7f62bc1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 >> v1:172.21.15.96:6789/0 conn(0x7f62bc1430c0 legacy=0x7f62bc1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 1 monclient: found mon.c 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc1d84d0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 5 monclient: authenticate success, global_id 4415 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 10 monclient: _renew_subs 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc1d41e0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:38.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 10 monclient: _renew_subs 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc1d5150 con 0x7f62bc0a98f0 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.191+0000 7f62c46c8640 1 librados: init done 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.193+0000 7f62c46c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.193+0000 7f62c46c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.193+0000 7f62c46c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:38.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.193+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc0627b0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.199+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b8001c70 con 0x7f62bc0a98f0 2024-03-25T17:33:38.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.199+0000 7f62b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:38.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.199+0000 7f62c0c61640 4 set_mon_vals no callback set 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.202+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f62b8003a60 con 0x7f62bc0a98f0 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.202+0000 7f62b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.202+0000 7f62b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.202+0000 7f62b2ffd640 10 monclient: dump: 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:38.202 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:38.203 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:38.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.222+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f62b8003c40 con 0x7f62bc0a98f0 2024-03-25T17:33:38.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.223+0000 7f62c1c63640 10 monclient: get_auth_request con 0x7f62a006de40 auth_method 0 2024-03-25T17:33:38.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.223+0000 7f62b2ffd640 10 client.4415.objecter ms_handle_connect 0x7f62a006de40 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f62b80548c0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 10 client.4415.objecter ms_dispatch 0x7f62bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62b80503d0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:38.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.503+0000 7f62b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:38.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.757+0000 7f62c46c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:38.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.757+0000 7f62c46c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:38.757 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.757+0000 7f62c46c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:38.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.757+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62bc074cc0 con 0x7f62bc0a98f0 2024-03-25T17:33:38.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.782+0000 7f62b2ffd640 1 -- 172.21.15.96:0/1396768897 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f62b8052300 con 0x7f62bc0a98f0 2024-03-25T17:33:38.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.782+0000 7f62b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:38.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.782+0000 7f62b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:38.783 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:38.784 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 10 librados: watch_flush enter 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 10 librados: watch_flush exit 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 10 client.4415.objecter successfully canceled tick 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 >> v1:172.21.15.203:6800/3611964061 conn(0x7f62a006de40 legacy=0x7f62a0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 10 monclient: shutdown 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 >> v1:172.21.15.96:6790/0 conn(0x7f62bc0a98f0 legacy=0x7f62bc137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:38.786 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 shutdown_connections 2024-03-25T17:33:38.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 >> 172.21.15.96:0/1396768897 conn(0x7f62bc0b7ea0 msgr2=0x7f62bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:38.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.785+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 shutdown_connections 2024-03-25T17:33:38.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.786+0000 7f62c46c8640 1 -- 172.21.15.96:0/1396768897 wait complete. 2024-03-25T17:33:38.787 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:38.786+0000 7f62c46c8640 1 librados: shutdown 2024-03-25T17:33:38.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:40.671 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:04.898 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.688 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:04.914 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.716 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:04.942 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.736 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:04.963 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.742 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:04.968 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.760 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:04.975 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.763 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:04.979 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.777 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:05.003 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:40.795 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:05.010 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:45.605 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.729+0000 7f94f0b3e640 10 monclient: get_monmap_and_config 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.729+0000 7f94f0b3e640 10 monclient: build_initial_monmap 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.729+0000 7f94f0b3e640 10 monclient: monmap: 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:45.730481+0000 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:45.730481+0000 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:45.730 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:45.731 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:45.731 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.731 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.729+0000 7f94f0b3e640 1 Processor -- start 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 1 -- start start 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient: init 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f94ec0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f94ec13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f94ec1430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.730+0000 7f94f0b3e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec142520 con 0x7f94ec1430c0 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.731+0000 7f94f0b3e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b28a0 con 0x7f94ec13f710 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.731+0000 7f94f0b3e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b2a30 con 0x7f94ec0a98f0 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.731+0000 7f94f0b3e640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:45.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.731+0000 7f94f0b3e640 10 monclient(hunting): authenticate will time out at 5584.012966s 2024-03-25T17:33:45.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.738+0000 7f94e9d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f94ec13f710 0x7f94ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:55286/0 (socket says 172.21.15.96:55286) 2024-03-25T17:33:45.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.738+0000 7f94e9d9b640 1 -- 172.21.15.96:0/634756396 learned_addr learned my addr 172.21.15.96:0/634756396 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:45.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052017646 0 0) 0x7f94ec142520 con 0x7f94ec1430c0 2024-03-25T17:33:45.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 10 monclient(hunting): _negotiate 2024-03-25T17:33:45.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:45.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:45.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 10 monclient(hunting): my global_id is 4431 2024-03-25T17:33:45.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.751+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d4003520 con 0x7f94ec1430c0 2024-03-25T17:33:45.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039399875 0 0) 0x7f94d4003520 con 0x7f94ec1430c0 2024-03-25T17:33:45.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 >> v1:172.21.15.96:6790/0 conn(0x7f94ec0a98f0 legacy=0x7f94ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:45.758 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 >> v1:172.21.15.203:6789/0 conn(0x7f94ec13f710 legacy=0x7f94ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: _finish_hunting 0 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 monclient: found mon.noname-a 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0b44c0 con 0x7f94ec1430c0 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: _finish_auth 0 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: _check_auth_tickets 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e4001a10 con 0x7f94ec1430c0 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94e959a640 10 monclient: dump: 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:45.759 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94f0b3e640 5 monclient: authenticate success, global_id 4431 2024-03-25T17:33:45.760 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.757+0000 7f94f0b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:45.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001be0 con 0x7f94ec1430c0 2024-03-25T17:33:45.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:45.769 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 1 -- 172.21.15.96:0/634756396 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e4002030 con 0x7f94ec1430c0 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94e959a640 10 monclient: dump: 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:45.770 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 4 set_mon_vals no callback set 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 10 monclient: shutdown 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 1 -- 172.21.15.96:0/634756396 >> v1:172.21.15.96:6789/0 conn(0x7f94ec1430c0 legacy=0x7f94ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 1 -- 172.21.15.96:0/634756396 shutdown_connections 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 1 -- 172.21.15.96:0/634756396 >> 172.21.15.96:0/634756396 conn(0x7f94ec0b7ea0 msgr2=0x7f94ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 1 -- 172.21.15.96:0/634756396 shutdown_connections 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.769+0000 7f94f0b3e640 1 -- 172.21.15.96:0/634756396 wait complete. 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.770+0000 7f94f0b3e640 10 monclient: build_initial_monmap 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.770+0000 7f94f0b3e640 10 monclient: monmap: 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:45.772 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 librados: starting msgr at 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 librados: starting objecter 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 Processor -- start 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 -- start start 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 librados: setting wanted keys 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 1 librados: calling monclient init 2024-03-25T17:33:45.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 10 monclient: init 2024-03-25T17:33:45.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:45.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.771+0000 7f94f0b3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:33:45.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f94ec0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f94ec13f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f94ec1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec136cc0 con 0x7f94ec13f710 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec136fa0 con 0x7f94ec1430c0 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec137280 con 0x7f94ec0a98f0 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:45.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.772+0000 7f94f0b3e640 10 monclient(hunting): authenticate will time out at 5584.054130s 2024-03-25T17:33:45.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.778+0000 7f94e9d9b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f94ec13f710 0x7f94ec1376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51368/0 (socket says 172.21.15.96:51368) 2024-03-25T17:33:45.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.778+0000 7f94e9d9b640 1 -- 172.21.15.96:0/3718839882 learned_addr learned my addr 172.21.15.96:0/3718839882 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:45.779 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.779+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec13f710 2024-03-25T17:33:45.779 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.779+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:45.785 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.784+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec1430c0 2024-03-25T17:33:45.785 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.784+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764919231 0 0) 0x7f94ec136cc0 con 0x7f94ec13f710 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 10 monclient(hunting): my global_id is 4434 2024-03-25T17:33:45.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.788+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc004eb0 con 0x7f94ec13f710 2024-03-25T17:33:45.790 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.789+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec0a98f0 2024-03-25T17:33:45.790 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.789+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:45.791 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054865123 0 0) 0x7f94ec136fa0 con 0x7f94ec1430c0 2024-03-25T17:33:45.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:45.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:45.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:45.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 10 monclient(hunting): my global_id is 4420 2024-03-25T17:33:45.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.791+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec136cc0 con 0x7f94ec1430c0 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 637890248 0 0) 0x7f94ec136cc0 con 0x7f94ec1430c0 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec136fa0 con 0x7f94ec1430c0 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e4001ef0 con 0x7f94ec1430c0 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 10 monclient(hunting): dump: 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:45.808 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.807+0000 7f94daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077688538 0 0) 0x7f94cc004eb0 con 0x7f94ec13f710 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec136cc0 con 0x7f94ec13f710 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e00018b0 con 0x7f94ec13f710 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.815+0000 7f94daffd640 10 monclient(hunting): dump: 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:45.816 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:45.817 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:45.817 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:45.817 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1464061067 0 0) 0x7f94ec136fa0 con 0x7f94ec1430c0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 >> v1:172.21.15.96:6790/0 conn(0x7f94ec0a98f0 legacy=0x7f94ec135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 >> v1:172.21.15.96:6789/0 conn(0x7f94ec13f710 legacy=0x7f94ec1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 1 monclient: found mon.b 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec13bbf0 con 0x7f94ec1430c0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 5 monclient: authenticate success, global_id 4420 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 10 monclient: _renew_subs 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:45.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec135f40 con 0x7f94ec1430c0 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 10 monclient: _renew_subs 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec1e34b0 con 0x7f94ec1430c0 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.816+0000 7f94f0b3e640 1 librados: init done 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.817+0000 7f94f0b3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.817+0000 7f94f0b3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.817+0000 7f94f0b3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:45.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:45.817+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec0627b0 con 0x7f94ec1430c0 2024-03-25T17:33:46.102 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e4002090 con 0x7f94ec1430c0 2024-03-25T17:33:46.102 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e4003dc0 con 0x7f94ec1430c0 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94daffd640 10 monclient: dump: 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:46.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.101+0000 7f94e8d99640 4 set_mon_vals no callback set 2024-03-25T17:33:46.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.102+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f94e4019d90 con 0x7f94ec1430c0 2024-03-25T17:33:46.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.103+0000 7f94ea59c640 10 monclient: get_auth_request con 0x7f94cc06de40 auth_method 0 2024-03-25T17:33:46.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.103+0000 7f94daffd640 10 client.4420.objecter ms_handle_connect 0x7f94cc06de40 2024-03-25T17:33:46.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.107+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f94e4053900 con 0x7f94ec1430c0 2024-03-25T17:33:46.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.107+0000 7f94daffd640 10 client.4420.objecter ms_dispatch 0x7f94ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:46.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.107+0000 7f94daffd640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:46.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.107+0000 7f94daffd640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:46.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.107+0000 7f94daffd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T17:33:46.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.108+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94e404f410 con 0x7f94ec1430c0 2024-03-25T17:33:46.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.108+0000 7f94daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:46.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.108+0000 7f94daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:46.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.350+0000 7f94f0b3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:46.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.350+0000 7f94f0b3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:46.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.350+0000 7f94f0b3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:33:46.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.350+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94ec13df50 con 0x7f94ec1430c0 2024-03-25T17:33:46.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.373+0000 7f94daffd640 1 -- 172.21.15.96:0/3718839882 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f94e4051340 con 0x7f94ec1430c0 2024-03-25T17:33:46.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.373+0000 7f94daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:46.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.373+0000 7f94daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:46.375 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:46.375 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:46.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 10 librados: watch_flush enter 2024-03-25T17:33:46.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 10 librados: watch_flush exit 2024-03-25T17:33:46.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 10 client.4420.objecter successfully canceled tick 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 >> v1:172.21.15.203:6800/3611964061 conn(0x7f94cc06de40 legacy=0x7f94cc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 10 monclient: shutdown 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 >> v1:172.21.15.203:6789/0 conn(0x7f94ec1430c0 legacy=0x7f94ec0b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 shutdown_connections 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 >> 172.21.15.96:0/3718839882 conn(0x7f94ec0b7ea0 msgr2=0x7f94ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 shutdown_connections 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 -- 172.21.15.96:0/3718839882 wait complete. 2024-03-25T17:33:46.378 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:46.376+0000 7f94f0b3e640 1 librados: shutdown 2024-03-25T17:33:46.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:53.196 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:53.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 10 monclient: get_monmap_and_config 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 10 monclient: build_initial_monmap 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 10 monclient: monmap: 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:33:53.308321+0000 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:33:53.308321+0000 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:33:53.308 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:33:53.309 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 1 Processor -- start 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 1 -- start start 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.307+0000 7f093938f640 10 monclient: init 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient(hunting): picked mon.noname-b con 0x7f09340a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient(hunting): picked mon.noname-c con 0x7f093413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient(hunting): picked mon.noname-a con 0x7f09341430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0934142520 con 0x7f09341430c0 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340b28a0 con 0x7f093413f710 2024-03-25T17:33:53.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340b2a30 con 0x7f09340a98f0 2024-03-25T17:33:53.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:53.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.308+0000 7f093938f640 10 monclient(hunting): authenticate will time out at 5591.590853s 2024-03-25T17:33:53.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.318+0000 7f0932ffd640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f09340a98f0 0x7f09340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:38530/0 (socket says 172.21.15.96:38530) 2024-03-25T17:33:53.319 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.318+0000 7f0932ffd640 1 -- 172.21.15.96:0/1935253043 learned_addr learned my addr 172.21.15.96:0/1935253043 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:53.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617533769 0 0) 0x7f09340b28a0 con 0x7f093413f710 2024-03-25T17:33:53.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T17:33:53.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.337+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c003520 con 0x7f093413f710 2024-03-25T17:33:53.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851508473 0 0) 0x7f09340b2a30 con 0x7f09340a98f0 2024-03-25T17:33:53.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T17:33:53.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.339+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09340b28a0 con 0x7f09340a98f0 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995821994 0 0) 0x7f0934142520 con 0x7f09341430c0 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09340b2a30 con 0x7f09341430c0 2024-03-25T17:33:53.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626372546 0 0) 0x7f09340b28a0 con 0x7f09340a98f0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 >> v1:172.21.15.203:6789/0 conn(0x7f093413f710 legacy=0x7f09341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 >> v1:172.21.15.96:6789/0 conn(0x7f09341430c0 legacy=0x7f093414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 monclient: found mon.noname-b 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340b44c0 con 0x7f09340a98f0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: _finish_auth 0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0924001670 con 0x7f09340a98f0 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f093938f640 5 monclient: authenticate success, global_id 4424 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f093938f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:53.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f0931ffb640 10 monclient: dump: 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.356+0000 7f093938f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09240017d0 con 0x7f09340a98f0 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 1 -- 172.21.15.96:0/1935253043 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0924001be0 con 0x7f09340a98f0 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f093938f640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f093938f640 4 set_mon_vals no callback set 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.369+0000 7f0931ffb640 10 monclient: dump: 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.370 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.371 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 10 monclient: shutdown 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 1 -- 172.21.15.96:0/1935253043 >> v1:172.21.15.96:6790/0 conn(0x7f09340a98f0 legacy=0x7f09340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 1 -- 172.21.15.96:0/1935253043 shutdown_connections 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 1 -- 172.21.15.96:0/1935253043 >> 172.21.15.96:0/1935253043 conn(0x7f09340b7ea0 msgr2=0x7f09340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 1 -- 172.21.15.96:0/1935253043 shutdown_connections 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.370+0000 7f093938f640 1 -- 172.21.15.96:0/1935253043 wait complete. 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 10 monclient: build_initial_monmap 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 10 monclient: monmap: 2024-03-25T17:33:53.373 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 librados: starting msgr at 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 librados: starting objecter 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 Processor -- start 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 -- start start 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 librados: setting wanted keys 2024-03-25T17:33:53.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 1 librados: calling monclient init 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.371+0000 7f093938f640 10 monclient: init 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient(hunting): picked mon.noname-c con 0x7f09340a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient(hunting): picked mon.noname-a con 0x7f093413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient(hunting): picked mon.noname-b con 0x7f09341430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0934136e60 con 0x7f093413f710 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09341381a0 con 0x7f09341430c0 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09341370b0 con 0x7f09340a98f0 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:53.375 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.372+0000 7f093938f640 10 monclient(hunting): authenticate will time out at 5591.654442s 2024-03-25T17:33:53.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.381+0000 7f09327fc640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f093413f710 0x7f093413a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56270/0 (socket says 172.21.15.96:56270) 2024-03-25T17:33:53.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.381+0000 7f09327fc640 1 -- 172.21.15.96:0/3614715924 learned_addr learned my addr 172.21.15.96:0/3614715924 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:33:53.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.382+0000 7f091b7fe640 10 client.?.objecter ms_handle_connect 0x7f093413f710 2024-03-25T17:33:53.383 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.382+0000 7f091b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:53.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.393+0000 7f091b7fe640 10 client.?.objecter ms_handle_connect 0x7f09340a98f0 2024-03-25T17:33:53.394 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.393+0000 7f091b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:53.397 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.397+0000 7f091b7fe640 10 client.?.objecter ms_handle_connect 0x7f09341430c0 2024-03-25T17:33:53.398 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.397+0000 7f091b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099981372 0 0) 0x7f09341370b0 con 0x7f09340a98f0 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T17:33:53.404 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.403+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920004eb0 con 0x7f09340a98f0 2024-03-25T17:33:53.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481072797 0 0) 0x7f0934136e60 con 0x7f093413f710 2024-03-25T17:33:53.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.405 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 10 monclient(hunting): my global_id is 4440 2024-03-25T17:33:53.406 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.405+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09341370b0 con 0x7f093413f710 2024-03-25T17:33:53.408 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798730755 0 0) 0x7f09341381a0 con 0x7f09341430c0 2024-03-25T17:33:53.408 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:53.408 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:53.408 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:53.408 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T17:33:53.409 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.407+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0934136e60 con 0x7f09341430c0 2024-03-25T17:33:53.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900183209 0 0) 0x7f0920004eb0 con 0x7f09340a98f0 2024-03-25T17:33:53.414 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09341381a0 con 0x7f09340a98f0 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0924001ae0 con 0x7f09340a98f0 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.415 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.413+0000 7f091b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220277763 0 0) 0x7f09341370b0 con 0x7f093413f710 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0920004eb0 con 0x7f093413f710 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0928001950 con 0x7f093413f710 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.414+0000 7f091b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.416 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.417 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210450924 0 0) 0x7f0934136e60 con 0x7f09341430c0 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09341370b0 con 0x7f09341430c0 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f092c001b10 con 0x7f09341430c0 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.418+0000 7f091b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.419 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.420 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.420 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.420 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 916229659 0 0) 0x7f09341381a0 con 0x7f09340a98f0 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 >> v1:172.21.15.203:6789/0 conn(0x7f09341430c0 legacy=0x7f093413ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 >> v1:172.21.15.96:6789/0 conn(0x7f093413f710 legacy=0x7f093413a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 1 monclient: found mon.c 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0934137390 con 0x7f09340a98f0 2024-03-25T17:33:53.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f091b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f093938f640 5 monclient: authenticate success, global_id 4427 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f093938f640 10 monclient: _renew_subs 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f093938f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.419+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09341376d0 con 0x7f09340a98f0 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 monclient: _renew_subs 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0934135530 con 0x7f09340a98f0 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 1 librados: init done 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:53.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.423 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.420+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09340b3ef0 con 0x7f09340a98f0 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0924004a90 con 0x7f09340a98f0 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0924004da0 con 0x7f09340a98f0 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f091b7fe640 10 monclient: dump: 2024-03-25T17:33:53.727 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:33:53.728 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.726+0000 7f09317fa640 4 set_mon_vals no callback set 2024-03-25T17:33:53.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.727+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f092401ad70 con 0x7f09340a98f0 2024-03-25T17:33:53.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.728+0000 7f09327fc640 10 monclient: get_auth_request con 0x7f092006e0b0 auth_method 0 2024-03-25T17:33:53.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.728+0000 7f091b7fe640 10 client.4427.objecter ms_handle_connect 0x7f092006e0b0 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f0924054860 con 0x7f09340a98f0 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 10 client.4427.objecter ms_dispatch 0x7f0934000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09240501a0 con 0x7f09340a98f0 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:53.734 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.733+0000 7f091b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:53.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.992+0000 7f093938f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:53.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.992+0000 7f093938f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:53.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.992+0000 7f093938f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:33:53.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:53.992+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09341357e0 con 0x7f09340a98f0 2024-03-25T17:33:54.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.038+0000 7f091b7fe640 1 -- 172.21.15.96:0/3614715924 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f09240522a0 con 0x7f09340a98f0 2024-03-25T17:33:54.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.038+0000 7f091b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:54.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.038+0000 7f091b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:54.039 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:33:54.040 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:54.041 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 10 librados: watch_flush enter 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 10 librados: watch_flush exit 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 10 client.4427.objecter successfully canceled tick 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 >> v1:172.21.15.203:6800/3611964061 conn(0x7f092006e0b0 legacy=0x7f0920070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 10 monclient: shutdown 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 >> v1:172.21.15.96:6790/0 conn(0x7f09340a98f0 legacy=0x7f0934137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 shutdown_connections 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 >> 172.21.15.96:0/3614715924 conn(0x7f09340b7ea0 msgr2=0x7f0934074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:54.042 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.040+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 shutdown_connections 2024-03-25T17:33:54.043 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.041+0000 7f093938f640 1 -- 172.21.15.96:0/3614715924 wait complete. 2024-03-25T17:33:54.043 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:33:54.041+0000 7f093938f640 1 librados: shutdown 2024-03-25T17:33:54.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:00.663 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:24.890 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.708 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:24.935 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.708 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:24.923 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.710 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:24.926 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.721 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:24.936 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.734 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:24.961 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.850 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:25.076 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.860 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:00.868 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:25.094 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.898 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:25.124 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.969+0000 7f773079d640 10 monclient: get_monmap_and_config 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.969+0000 7f773079d640 10 monclient: build_initial_monmap 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.969+0000 7f773079d640 10 monclient: monmap: 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:00.971044+0000 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:00.971044+0000 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:00.970 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.970+0000 7f773079d640 1 Processor -- start 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.970+0000 7f773079d640 1 -- start start 2024-03-25T17:34:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.970+0000 7f773079d640 10 monclient: init 2024-03-25T17:34:00.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient(hunting): picked mon.noname-b con 0x7f77280a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient(hunting): picked mon.noname-a con 0x7f772813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient(hunting): picked mon.noname-c con 0x7f77281430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7728142520 con 0x7f772813f710 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280b28a0 con 0x7f77281430c0 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280b2a30 con 0x7f77280a98f0 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:00.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.971+0000 7f773079d640 10 monclient(hunting): authenticate will time out at 5599.253822s 2024-03-25T17:34:00.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.978+0000 7f772ed3a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f77281430c0 0x7f772814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:43896/0 (socket says 172.21.15.96:43896) 2024-03-25T17:34:00.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.978+0000 7f772ed3a640 1 -- 172.21.15.96:0/2453828403 learned_addr learned my addr 172.21.15.96:0/2453828403 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:00.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096380352 0 0) 0x7f77280b28a0 con 0x7f77281430c0 2024-03-25T17:34:00.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 10 monclient(hunting): _negotiate 2024-03-25T17:34:00.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:00.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:00.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 10 monclient(hunting): my global_id is 4429 2024-03-25T17:34:00.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.991+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7714003520 con 0x7f77281430c0 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308096096 0 0) 0x7f7728142520 con 0x7f772813f710 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 10 monclient(hunting): _negotiate 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 10 monclient(hunting): my global_id is 4443 2024-03-25T17:34:00.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:00.993+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77280b28a0 con 0x7f772813f710 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.001+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688897479 0 0) 0x7f77280b28a0 con 0x7f772813f710 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 >> v1:172.21.15.96:6790/0 conn(0x7f77280a98f0 legacy=0x7f77280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 >> v1:172.21.15.203:6789/0 conn(0x7f77281430c0 legacy=0x7f772814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: _finish_hunting 0 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 1 monclient: found mon.noname-a 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280b44c0 con 0x7f772813f710 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: _finish_auth 0 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: _check_auth_tickets 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7718001670 con 0x7f772813f710 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:01.003 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f772d537640 10 monclient: dump: 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f773079d640 5 monclient: authenticate success, global_id 4443 2024-03-25T17:34:01.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.002+0000 7f773079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77180017d0 con 0x7f772813f710 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f773079d640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f773079d640 4 set_mon_vals no callback set 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 1 -- 172.21.15.96:0/2453828403 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7718001be0 con 0x7f772813f710 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.016+0000 7f772d537640 10 monclient: dump: 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:01.017 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:01.018 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 10 monclient: shutdown 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 1 -- 172.21.15.96:0/2453828403 >> v1:172.21.15.96:6789/0 conn(0x7f772813f710 legacy=0x7f77281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 1 -- 172.21.15.96:0/2453828403 shutdown_connections 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 1 -- 172.21.15.96:0/2453828403 >> 172.21.15.96:0/2453828403 conn(0x7f77280b7ea0 msgr2=0x7f77280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 1 -- 172.21.15.96:0/2453828403 shutdown_connections 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 1 -- 172.21.15.96:0/2453828403 wait complete. 2024-03-25T17:34:01.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 10 monclient: build_initial_monmap 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.017+0000 7f773079d640 10 monclient: monmap: 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 librados: starting msgr at 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 librados: starting objecter 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 Processor -- start 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 -- start start 2024-03-25T17:34:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 librados: setting wanted keys 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 1 librados: calling monclient init 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.018+0000 7f773079d640 10 monclient: init 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient(hunting): picked mon.noname-a con 0x7f77280a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient(hunting): picked mon.noname-b con 0x7f772813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient(hunting): picked mon.noname-c con 0x7f77281430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77281398c0 con 0x7f77280a98f0 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280abb00 con 0x7f772813f710 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280abde0 con 0x7f77281430c0 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:01.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.019+0000 7f773079d640 10 monclient(hunting): authenticate will time out at 5599.301165s 2024-03-25T17:34:01.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.031+0000 7f772dd38640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f772813f710 0x7f77281df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:43954/0 (socket says 172.21.15.96:43954) 2024-03-25T17:34:01.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.032+0000 7f772dd38640 1 -- 172.21.15.96:0/3326297983 learned_addr learned my addr 172.21.15.96:0/3326297983 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:01.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.033+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f772813f710 2024-03-25T17:34:01.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.033+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:01.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.048+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f77280a98f0 2024-03-25T17:34:01.049 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.048+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:01.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.051+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f77281430c0 2024-03-25T17:34:01.052 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.051+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099731870 0 0) 0x7f77280abb00 con 0x7f772813f710 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 10 monclient(hunting): my global_id is 4432 2024-03-25T17:34:01.055 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.054+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f770c004eb0 con 0x7f772813f710 2024-03-25T17:34:01.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1320673974 0 0) 0x7f770c004eb0 con 0x7f772813f710 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77280abb00 con 0x7f772813f710 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7718001be0 con 0x7f772813f710 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 10 monclient(hunting): dump: 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:01.064 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:01.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.063+0000 7f771effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:01.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601267613 0 0) 0x7f77281398c0 con 0x7f77280a98f0 2024-03-25T17:34:01.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:01.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:01.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:01.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 10 monclient(hunting): my global_id is 4446 2024-03-25T17:34:01.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.079+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f770c004eb0 con 0x7f77280a98f0 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487399776 0 0) 0x7f77280abde0 con 0x7f77281430c0 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 10 monclient(hunting): my global_id is 4433 2024-03-25T17:34:01.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.087+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77281398c0 con 0x7f77281430c0 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2900001763 0 0) 0x7f77280abb00 con 0x7f772813f710 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 >> v1:172.21.15.96:6790/0 conn(0x7f77281430c0 legacy=0x7f77281e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 >> v1:172.21.15.96:6789/0 conn(0x7f77280a98f0 legacy=0x7f7728135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 1 monclient: found mon.b 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772814e550 con 0x7f772813f710 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 10 monclient: _finish_auth 0 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.093+0000 7f771effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 5 monclient: authenticate success, global_id 4432 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 10 monclient: _renew_subs 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:01.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7728137a90 con 0x7f772813f710 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 10 monclient: _renew_subs 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77280ac3e0 con 0x7f772813f710 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.094+0000 7f773079d640 1 librados: init done 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.095+0000 7f773079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:01.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.095+0000 7f773079d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:01.097 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.095+0000 7f773079d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:01.097 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.095+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77280b3ef0 con 0x7f772813f710 2024-03-25T17:34:01.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77180038a0 con 0x7f772813f710 2024-03-25T17:34:01.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:01.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7718003cb0 con 0x7f772813f710 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f771effd640 10 monclient: dump: 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:01.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.378+0000 7f772cd36640 4 set_mon_vals no callback set 2024-03-25T17:34:01.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.379+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7718003ed0 con 0x7f772813f710 2024-03-25T17:34:01.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.380+0000 7f772e539640 10 monclient: get_auth_request con 0x7f770c06dea0 auth_method 0 2024-03-25T17:34:01.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.380+0000 7f771effd640 10 client.4432.objecter ms_handle_connect 0x7f770c06dea0 2024-03-25T17:34:01.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f7718054930 con 0x7f772813f710 2024-03-25T17:34:01.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 10 client.4432.objecter ms_dispatch 0x7f7728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7718050440 con 0x7f772813f710 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:01.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.384+0000 7f771effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:01.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.633+0000 7f773079d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:01.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.633+0000 7f773079d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:01.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.633+0000 7f773079d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:01.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.633+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f772814e6e0 con 0x7f772813f710 2024-03-25T17:34:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.671+0000 7f771effd640 1 -- 172.21.15.96:0/3326297983 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f7718052370 con 0x7f772813f710 2024-03-25T17:34:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.671+0000 7f771effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.671+0000 7f771effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:01.672 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:01.672 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:01.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 10 librados: watch_flush enter 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 10 librados: watch_flush exit 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 10 client.4432.objecter successfully canceled tick 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 >> v1:172.21.15.203:6800/3611964061 conn(0x7f770c06dea0 legacy=0x7f770c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 10 monclient: shutdown 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 >> v1:172.21.15.203:6789/0 conn(0x7f772813f710 legacy=0x7f77281df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 shutdown_connections 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 >> 172.21.15.96:0/3326297983 conn(0x7f77280b7ea0 msgr2=0x7f7728074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 shutdown_connections 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 -- 172.21.15.96:0/3326297983 wait complete. 2024-03-25T17:34:01.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:01.673+0000 7f773079d640 1 librados: shutdown 2024-03-25T17:34:01.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:08.493 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.608+0000 7f3c6ae57640 10 monclient: get_monmap_and_config 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.608+0000 7f3c6ae57640 10 monclient: build_initial_monmap 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.608+0000 7f3c6ae57640 10 monclient: monmap: 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:08.609538+0000 2024-03-25T17:34:08.609 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:08.609538+0000 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:08.610 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 1 Processor -- start 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 1 -- start start 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient: init 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-c con 0x7f3c640a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-b con 0x7f3c641430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64142520 con 0x7f3c6413f710 2024-03-25T17:34:08.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b28a0 con 0x7f3c640a98f0 2024-03-25T17:34:08.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b2a30 con 0x7f3c641430c0 2024-03-25T17:34:08.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:08.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.609+0000 7f3c6ae57640 10 monclient(hunting): authenticate will time out at 5606.891865s 2024-03-25T17:34:08.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.619+0000 7f3c68bf3640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c640a98f0 0x7f3c640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33672/0 (socket says 172.21.15.96:33672) 2024-03-25T17:34:08.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.619+0000 7f3c68bf3640 1 -- 172.21.15.96:0/2149060899 learned_addr learned my addr 172.21.15.96:0/2149060899 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907770839 0 0) 0x7f3c640b28a0 con 0x7f3c640a98f0 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 10 monclient(hunting): my global_id is 4435 2024-03-25T17:34:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.626+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003520 con 0x7f3c640a98f0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 374999773 0 0) 0x7f3c54003520 con 0x7f3c640a98f0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 >> v1:172.21.15.96:6790/0 conn(0x7f3c641430c0 legacy=0x7f3c6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 >> v1:172.21.15.96:6789/0 conn(0x7f3c6413f710 legacy=0x7f3c641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 10 monclient: _finish_hunting 0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 monclient: found mon.noname-c 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640b44c0 con 0x7f3c640a98f0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 10 monclient: _finish_auth 0 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 10 monclient: _check_auth_tickets 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:08.643 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c58001690 con 0x7f3c640a98f0 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c6ae57640 5 monclient: authenticate success, global_id 4435 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c6ae57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.640+0000 7f3c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.641+0000 7f3c637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.641+0000 7f3c637fe640 10 monclient: dump: 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:08.644 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:08.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.641+0000 7f3c6ae57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c580017f0 con 0x7f3c640a98f0 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 1 -- 172.21.15.96:0/2149060899 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c58001c00 con 0x7f3c640a98f0 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c637fe640 10 monclient: dump: 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:08.646 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:08.647 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:08.647 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:08.647 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:08.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c6ae57640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:08.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c6ae57640 4 set_mon_vals no callback set 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c6ae57640 10 monclient: shutdown 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c6ae57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.645+0000 7f3c6ae57640 1 -- 172.21.15.96:0/2149060899 >> v1:172.21.15.203:6789/0 conn(0x7f3c640a98f0 legacy=0x7f3c640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 1 -- 172.21.15.96:0/2149060899 shutdown_connections 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 1 -- 172.21.15.96:0/2149060899 >> 172.21.15.96:0/2149060899 conn(0x7f3c640b7ea0 msgr2=0x7f3c640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 1 -- 172.21.15.96:0/2149060899 shutdown_connections 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 1 -- 172.21.15.96:0/2149060899 wait complete. 2024-03-25T17:34:08.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 10 monclient: build_initial_monmap 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.646+0000 7f3c6ae57640 10 monclient: monmap: 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 librados: starting msgr at 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 librados: starting objecter 2024-03-25T17:34:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 Processor -- start 2024-03-25T17:34:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 -- start start 2024-03-25T17:34:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 librados: setting wanted keys 2024-03-25T17:34:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 1 librados: calling monclient init 2024-03-25T17:34:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.647+0000 7f3c6ae57640 10 monclient: init 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-a con 0x7f3c640a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient(hunting): picked mon.noname-c con 0x7f3c641430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c641db7d0 con 0x7f3c640a98f0 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b1420 con 0x7f3c6413f710 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b15b0 con 0x7f3c641430c0 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:08.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.648+0000 7f3c6ae57640 10 monclient(hunting): authenticate will time out at 5606.930543s 2024-03-25T17:34:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.662+0000 7f3c63fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c6413f710 0x7f3c640b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33700/0 (socket says 172.21.15.96:33700) 2024-03-25T17:34:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.662+0000 7f3c63fff640 1 -- 172.21.15.96:0/3419741241 learned_addr learned my addr 172.21.15.96:0/3419741241 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:08.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.664+0000 7f3c617fa640 10 client.?.objecter ms_handle_connect 0x7f3c6413f710 2024-03-25T17:34:08.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.664+0000 7f3c617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:08.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.675+0000 7f3c617fa640 10 client.?.objecter ms_handle_connect 0x7f3c641430c0 2024-03-25T17:34:08.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.675+0000 7f3c617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:08.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.675+0000 7f3c617fa640 10 client.?.objecter ms_handle_connect 0x7f3c640a98f0 2024-03-25T17:34:08.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.675+0000 7f3c617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:08.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663758664 0 0) 0x7f3c640b15b0 con 0x7f3c641430c0 2024-03-25T17:34:08.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:08.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:08.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): my global_id is 4439 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48004eb0 con 0x7f3c641430c0 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721218278 0 0) 0x7f3c640b1420 con 0x7f3c6413f710 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 10 monclient(hunting): my global_id is 4438 2024-03-25T17:34:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.681+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c640b15b0 con 0x7f3c6413f710 2024-03-25T17:34:08.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867745485 0 0) 0x7f3c641db7d0 con 0x7f3c640a98f0 2024-03-25T17:34:08.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:08.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:08.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:08.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 10 monclient(hunting): my global_id is 4452 2024-03-25T17:34:08.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.686+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c640b1420 con 0x7f3c640a98f0 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010752620 0 0) 0x7f3c48004eb0 con 0x7f3c641430c0 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c641db7d0 con 0x7f3c641430c0 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c5c001b20 con 0x7f3c641430c0 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 10 monclient(hunting): dump: 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:08.699 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:08.700 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.698+0000 7f3c617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:08.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820273925 0 0) 0x7f3c641db7d0 con 0x7f3c641430c0 2024-03-25T17:34:08.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 >> v1:172.21.15.203:6789/0 conn(0x7f3c6413f710 legacy=0x7f3c640b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 >> v1:172.21.15.96:6789/0 conn(0x7f3c640a98f0 legacy=0x7f3c641daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 10 monclient: _finish_hunting 0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 1 monclient: found mon.c 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640b1740 con 0x7f3c641430c0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 10 monclient: _finish_auth 0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 10 monclient: _check_auth_tickets 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 5 monclient: authenticate success, global_id 4439 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 10 monclient: _renew_subs 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c641db1a0 con 0x7f3c641430c0 2024-03-25T17:34:08.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 10 monclient: _renew_subs 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c640b08f0 con 0x7f3c641430c0 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.703+0000 7f3c6ae57640 1 librados: init done 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.704+0000 7f3c6ae57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:08.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.704+0000 7f3c6ae57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:08.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.704+0000 7f3c6ae57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:34:08.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:08.704+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c640627b0 con 0x7f3c641430c0 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c002030 con 0x7f3c641430c0 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3c5c003b10 con 0x7f3c641430c0 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c617fa640 10 monclient: dump: 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:09.021 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.020+0000 7f3c62ffd640 4 set_mon_vals no callback set 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.021+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f3c5c003cf0 con 0x7f3c641430c0 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.021+0000 7f3c68bf3640 10 monclient: get_auth_request con 0x7f3c4806ddf0 auth_method 0 2024-03-25T17:34:09.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.021+0000 7f3c617fa640 10 client.4439.objecter ms_handle_connect 0x7f3c4806ddf0 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f3c5c0537e0 con 0x7f3c641430c0 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 10 client.4439.objecter ms_dispatch 0x7f3c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c5c04f2f0 con 0x7f3c641430c0 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:09.028 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.027+0000 7f3c617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:09.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.244+0000 7f3c6ae57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:09.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.244+0000 7f3c6ae57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:09.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.244+0000 7f3c6ae57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:34:09.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.244+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c640abb10 con 0x7f3c641430c0 2024-03-25T17:34:09.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.284+0000 7f3c617fa640 1 -- 172.21.15.96:0/3419741241 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f3c5c051220 con 0x7f3c641430c0 2024-03-25T17:34:09.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.284+0000 7f3c617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:09.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.284+0000 7f3c617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:09.285 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:09.285 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 10 librados: watch_flush enter 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 10 librados: watch_flush exit 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 10 client.4439.objecter successfully canceled tick 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 >> v1:172.21.15.203:6800/3611964061 conn(0x7f3c4806ddf0 legacy=0x7f3c480702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 10 monclient: shutdown 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 >> v1:172.21.15.96:6790/0 conn(0x7f3c641430c0 legacy=0x7f3c641d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:09.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 shutdown_connections 2024-03-25T17:34:09.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.286+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 >> 172.21.15.96:0/3419741241 conn(0x7f3c640b7ea0 msgr2=0x7f3c6413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:09.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.287+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 shutdown_connections 2024-03-25T17:34:09.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.287+0000 7f3c6ae57640 1 -- 172.21.15.96:0/3419741241 wait complete. 2024-03-25T17:34:09.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:09.287+0000 7f3c6ae57640 1 librados: shutdown 2024-03-25T17:34:09.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:16.106 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.224+0000 7f5cf1b64640 10 monclient: get_monmap_and_config 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.224+0000 7f5cf1b64640 10 monclient: build_initial_monmap 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.224+0000 7f5cf1b64640 10 monclient: monmap: 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:16.224898+0000 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:16.224898+0000 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.224 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.225+0000 7f5cf1b64640 1 Processor -- start 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.225+0000 7f5cf1b64640 1 -- start start 2024-03-25T17:34:16.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.225+0000 7f5cf1b64640 10 monclient: init 2024-03-25T17:34:16.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:16.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-a con 0x7f5cec0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-b con 0x7f5cec0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0af940 con 0x7f5cec0b4d60 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0b0a70 con 0x7f5cec14be90 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0b0d50 con 0x7f5cec0b1730 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:16.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.226+0000 7f5cf1b64640 10 monclient(hunting): authenticate will time out at 5614.507538s 2024-03-25T17:34:16.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.237+0000 7f5ceb7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5cec0b4d60 0x7f5cec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45980/0 (socket says 172.21.15.96:45980) 2024-03-25T17:34:16.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.237+0000 7f5ceb7fe640 1 -- 172.21.15.96:0/4045380060 learned_addr learned my addr 172.21.15.96:0/4045380060 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737837587 0 0) 0x7f5cec0af940 con 0x7f5cec0b4d60 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 10 monclient(hunting): my global_id is 4455 2024-03-25T17:34:16.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.244+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4003520 con 0x7f5cec0b4d60 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197869035 0 0) 0x7f5cd4003520 con 0x7f5cec0b4d60 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 >> v1:172.21.15.96:6790/0 conn(0x7f5cec0b1730 legacy=0x7f5cec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 >> v1:172.21.15.203:6789/0 conn(0x7f5cec14be90 legacy=0x7f5cec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 monclient: found mon.noname-a 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec0b1030 con 0x7f5cec0b4d60 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: _finish_auth 0 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:34:16.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cdc001690 con 0x7f5cec0b4d60 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cea7fc640 10 monclient: dump: 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:16.253 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:16.254 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cf1b64640 5 monclient: authenticate success, global_id 4455 2024-03-25T17:34:16.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.252+0000 7f5cf1b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cdc0017f0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 1 -- 172.21.15.96:0/4045380060 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cdc001c40 con 0x7f5cec0b4d60 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cf1b64640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cf1b64640 4 set_mon_vals no callback set 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.262+0000 7f5cea7fc640 10 monclient: dump: 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.262 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:16.263 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.263 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:16.263 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:16.263 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:16.263 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 10 monclient: shutdown 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 1 -- 172.21.15.96:0/4045380060 >> v1:172.21.15.96:6789/0 conn(0x7f5cec0b4d60 legacy=0x7f5cec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 1 -- 172.21.15.96:0/4045380060 shutdown_connections 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 1 -- 172.21.15.96:0/4045380060 >> 172.21.15.96:0/4045380060 conn(0x7f5cec0617a0 msgr2=0x7f5cec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 1 -- 172.21.15.96:0/4045380060 shutdown_connections 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 1 -- 172.21.15.96:0/4045380060 wait complete. 2024-03-25T17:34:16.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 10 monclient: build_initial_monmap 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.263+0000 7f5cf1b64640 10 monclient: monmap: 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 librados: starting msgr at 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 librados: starting objecter 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 Processor -- start 2024-03-25T17:34:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 -- start start 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 librados: setting wanted keys 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 1 librados: calling monclient init 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.264+0000 7f5cf1b64640 10 monclient: init 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-b con 0x7f5cec0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient(hunting): picked mon.noname-a con 0x7f5cec14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec13bde0 con 0x7f5cec14be90 2024-03-25T17:34:16.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec140f50 con 0x7f5cec0b4d60 2024-03-25T17:34:16.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec1410e0 con 0x7f5cec0b1730 2024-03-25T17:34:16.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:16.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.265+0000 7f5cf1b64640 10 monclient(hunting): authenticate will time out at 5614.546262s 2024-03-25T17:34:16.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.280+0000 7f5ceaffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5cec0b4d60 0x7f5cec1d7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:35748/0 (socket says 172.21.15.96:35748) 2024-03-25T17:34:16.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.280+0000 7f5ceaffd640 1 -- 172.21.15.96:0/796387620 learned_addr learned my addr 172.21.15.96:0/796387620 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:16.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.281+0000 7f5cd3fff640 10 client.?.objecter ms_handle_connect 0x7f5cec0b4d60 2024-03-25T17:34:16.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.281+0000 7f5cd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:16.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.282+0000 7f5cd3fff640 10 client.?.objecter ms_handle_connect 0x7f5cec0b1730 2024-03-25T17:34:16.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.282+0000 7f5cd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160171746 0 0) 0x7f5cec140f50 con 0x7f5cec0b4d60 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 10 monclient(hunting): my global_id is 4444 2024-03-25T17:34:16.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.288+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd8004eb0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.293+0000 7f5cd3fff640 10 client.?.objecter ms_handle_connect 0x7f5cec14be90 2024-03-25T17:34:16.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.293+0000 7f5cd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236488587 0 0) 0x7f5cec1410e0 con 0x7f5cec0b1730 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 10 monclient(hunting): my global_id is 4445 2024-03-25T17:34:16.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.301+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec140f50 con 0x7f5cec0b1730 2024-03-25T17:34:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386844395 0 0) 0x7f5cec13bde0 con 0x7f5cec14be90 2024-03-25T17:34:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:34:16.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:16.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:16.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 10 monclient(hunting): my global_id is 4458 2024-03-25T17:34:16.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.306+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec1410e0 con 0x7f5cec14be90 2024-03-25T17:34:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462578398 0 0) 0x7f5cd8004eb0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cec13bde0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5ce0001a30 con 0x7f5cec0b4d60 2024-03-25T17:34:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 10 monclient(hunting): dump: 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.308+0000 7f5cd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4248373369 0 0) 0x7f5cec140f50 con 0x7f5cec0b1730 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cd8004eb0 con 0x7f5cec0b1730 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cdc001b00 con 0x7f5cec0b1730 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.330+0000 7f5cd3fff640 10 monclient(hunting): dump: 2024-03-25T17:34:16.330 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:16.331 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2752266833 0 0) 0x7f5cec13bde0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 >> v1:172.21.15.96:6790/0 conn(0x7f5cec0b1730 legacy=0x7f5cec140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 >> v1:172.21.15.96:6789/0 conn(0x7f5cec14be90 legacy=0x7f5cec142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 1 monclient: found mon.b 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:16.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec1d8450 con 0x7f5cec0b4d60 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 10 monclient: _finish_auth 0 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 5 monclient: authenticate success, global_id 4444 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 10 monclient: _renew_subs 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec1d86a0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 10 monclient: _renew_subs 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec1426d0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.338 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.337+0000 7f5cf1b64640 1 librados: init done 2024-03-25T17:34:16.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.338+0000 7f5cf1b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:16.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.338+0000 7f5cf1b64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:16.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.338+0000 7f5cf1b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:16.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.338+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec1401f0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.617+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ce0001cf0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.617+0000 7f5cd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.617+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5ce0003a60 con 0x7f5cec0b4d60 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.618+0000 7f5cd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.618+0000 7f5cd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.618+0000 7f5ce9ffb640 4 set_mon_vals no callback set 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.618+0000 7f5cd3fff640 10 monclient: dump: 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:16.618 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.619+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5ce0003c40 con 0x7f5cec0b4d60 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.619+0000 7f5ceb7fe640 10 monclient: get_auth_request con 0x7f5cd806df70 auth_method 0 2024-03-25T17:34:16.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.619+0000 7f5cd3fff640 10 client.4444.objecter ms_handle_connect 0x7f5cd806df70 2024-03-25T17:34:16.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.623+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5ce00548e0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.623+0000 7f5cd3fff640 10 client.4444.objecter ms_dispatch 0x7f5cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:16.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.623+0000 7f5cd3fff640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:16.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.623+0000 7f5cd3fff640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:16.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.623+0000 7f5cd3fff640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T17:34:16.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.624+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ce00503f0 con 0x7f5cec0b4d60 2024-03-25T17:34:16.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.624+0000 7f5cd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:16.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.624+0000 7f5cd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:16.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.869+0000 7f5cf1b64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:16.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.869+0000 7f5cf1b64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:16.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.869+0000 7f5cf1b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:16.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.869+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cec1d7a60 con 0x7f5cec0b4d60 2024-03-25T17:34:16.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.969+0000 7f5cd3fff640 1 -- 172.21.15.96:0/796387620 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5cec1d7a60 con 0x7f5cec0b4d60 2024-03-25T17:34:16.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.969+0000 7f5cd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:16.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.969+0000 7f5cd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:16.969 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:16.969 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:16.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 10 librados: watch_flush enter 2024-03-25T17:34:16.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 10 librados: watch_flush exit 2024-03-25T17:34:16.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 10 client.4444.objecter successfully canceled tick 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 >> v1:172.21.15.203:6800/3611964061 conn(0x7f5cd806df70 legacy=0x7f5cd8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 10 monclient: shutdown 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 >> v1:172.21.15.203:6789/0 conn(0x7f5cec0b4d60 legacy=0x7f5cec1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 shutdown_connections 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 >> 172.21.15.96:0/796387620 conn(0x7f5cec0617a0 msgr2=0x7f5cec0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 shutdown_connections 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 -- 172.21.15.96:0/796387620 wait complete. 2024-03-25T17:34:16.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:16.971+0000 7f5cf1b64640 1 librados: shutdown 2024-03-25T17:34:16.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:20.775 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:45.001 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.781 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:45.007 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.820 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:05:45.046 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.835 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:45.050 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.837 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:45.063 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.841 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:45.067 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.847 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:45.062 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.872 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:05:45.098 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:20.880 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:45.095 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:23.789 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.902+0000 7f7c8e4f1640 10 monclient: get_monmap_and_config 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.902+0000 7f7c8e4f1640 10 monclient: build_initial_monmap 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.902+0000 7f7c8e4f1640 10 monclient: monmap: 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:23.903894+0000 2024-03-25T17:34:23.903 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:23.903894+0000 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:23.904 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.903+0000 7f7c8e4f1640 1 Processor -- start 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.903+0000 7f7c8e4f1640 1 -- start start 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.903+0000 7f7c8e4f1640 10 monclient: init 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:23.905 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c880a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c8813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c881430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88142520 con 0x7f7c880a98f0 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880b28a0 con 0x7f7c8813f710 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880b2a30 con 0x7f7c881430c0 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:23.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.904+0000 7f7c8e4f1640 10 monclient(hunting): authenticate will time out at 5622.186520s 2024-03-25T17:34:23.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.911+0000 7f7c877fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7c8813f710 0x7f7c881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:40560/0 (socket says 172.21.15.96:40560) 2024-03-25T17:34:23.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.911+0000 7f7c877fe640 1 -- 172.21.15.96:0/3987693418 learned_addr learned my addr 172.21.15.96:0/3987693418 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714701191 0 0) 0x7f7c880b28a0 con 0x7f7c8813f710 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 10 monclient(hunting): my global_id is 4447 2024-03-25T17:34:23.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.920+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c70003520 con 0x7f7c8813f710 2024-03-25T17:34:23.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954230222 0 0) 0x7f7c880b2a30 con 0x7f7c881430c0 2024-03-25T17:34:23.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:23.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:23.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:23.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 10 monclient(hunting): my global_id is 4448 2024-03-25T17:34:23.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.937+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c880b28a0 con 0x7f7c881430c0 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478429031 0 0) 0x7f7c70003520 con 0x7f7c8813f710 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 >> v1:172.21.15.96:6790/0 conn(0x7f7c881430c0 legacy=0x7f7c8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 >> v1:172.21.15.96:6789/0 conn(0x7f7c880a98f0 legacy=0x7f7c880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 monclient: found mon.noname-c 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880b44c0 con 0x7f7c8813f710 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: _finish_auth 0 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:23.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7c7c001690 con 0x7f7c8813f710 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c86ffd640 10 monclient: dump: 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c8e4f1640 5 monclient: authenticate success, global_id 4447 2024-03-25T17:34:23.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.938+0000 7f7c8e4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:23.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c7c0017f0 con 0x7f7c8813f710 2024-03-25T17:34:23.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:23.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 1 -- 172.21.15.96:0/3987693418 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7c7c001c00 con 0x7f7c8813f710 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c8e4f1640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c8e4f1640 4 set_mon_vals no callback set 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.956+0000 7f7c86ffd640 10 monclient: dump: 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:23.958 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:23.959 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:23.959 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 10 monclient: shutdown 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/3987693418 >> v1:172.21.15.203:6789/0 conn(0x7f7c8813f710 legacy=0x7f7c881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/3987693418 shutdown_connections 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/3987693418 >> 172.21.15.96:0/3987693418 conn(0x7f7c880b7ea0 msgr2=0x7f7c880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/3987693418 shutdown_connections 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.957+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/3987693418 wait complete. 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.958+0000 7f7c8e4f1640 10 monclient: build_initial_monmap 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.958+0000 7f7c8e4f1640 10 monclient: monmap: 2024-03-25T17:34:23.960 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 librados: starting msgr at 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 librados: starting objecter 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 Processor -- start 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 -- start start 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 librados: setting wanted keys 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 1 librados: calling monclient init 2024-03-25T17:34:23.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.959+0000 7f7c8e4f1640 10 monclient: init 2024-03-25T17:34:23.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c880a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c8813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c881430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88138c10 con 0x7f7c8813f710 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88138e60 con 0x7f7c880a98f0 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88139140 con 0x7f7c881430c0 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:23.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.960+0000 7f7c8e4f1640 10 monclient(hunting): authenticate will time out at 5622.242545s 2024-03-25T17:34:23.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.965+0000 7f7c877fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7c8813f710 0x7f7c88132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50738/0 (socket says 172.21.15.96:50738) 2024-03-25T17:34:23.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.965+0000 7f7c877fe640 1 -- 172.21.15.96:0/1789322369 learned_addr learned my addr 172.21.15.96:0/1789322369 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:23.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.966+0000 7f7c84ff9640 10 client.?.objecter ms_handle_connect 0x7f7c8813f710 2024-03-25T17:34:23.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.966+0000 7f7c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:23.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346001361 0 0) 0x7f7c88138c10 con 0x7f7c8813f710 2024-03-25T17:34:23.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:23.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:23.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:23.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 10 monclient(hunting): my global_id is 4464 2024-03-25T17:34:23.978 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.976+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c74004eb0 con 0x7f7c8813f710 2024-03-25T17:34:23.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.979+0000 7f7c84ff9640 10 client.?.objecter ms_handle_connect 0x7f7c880a98f0 2024-03-25T17:34:23.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.979+0000 7f7c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:23.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.981+0000 7f7c84ff9640 10 client.?.objecter ms_handle_connect 0x7f7c881430c0 2024-03-25T17:34:23.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.981+0000 7f7c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351653250 0 0) 0x7f7c74004eb0 con 0x7f7c8813f710 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c88138c10 con 0x7f7c8813f710 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7c7c001bc0 con 0x7f7c8813f710 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 10 monclient(hunting): dump: 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:23.985 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:23.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.984+0000 7f7c84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:23.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.986+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203914955 0 0) 0x7f7c88138e60 con 0x7f7c880a98f0 2024-03-25T17:34:23.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.986+0000 7f7c84ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:23.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.986+0000 7f7c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:23.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.986+0000 7f7c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:23.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.986+0000 7f7c84ff9640 10 monclient(hunting): my global_id is 4450 2024-03-25T17:34:23.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.987+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c74004eb0 con 0x7f7c880a98f0 2024-03-25T17:34:23.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117391597 0 0) 0x7f7c88139140 con 0x7f7c881430c0 2024-03-25T17:34:23.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:23.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:23.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:23.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 10 monclient(hunting): my global_id is 4451 2024-03-25T17:34:23.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.990+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c88138e60 con 0x7f7c881430c0 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.991+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180675055 0 0) 0x7f7c88138c10 con 0x7f7c8813f710 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 >> v1:172.21.15.96:6790/0 conn(0x7f7c881430c0 legacy=0x7f7c881e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 >> v1:172.21.15.203:6789/0 conn(0x7f7c880a98f0 legacy=0x7f7c88137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 1 monclient: found mon.a 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:23.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88139920 con 0x7f7c8813f710 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 10 monclient: _finish_auth 0 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 5 monclient: authenticate success, global_id 4464 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 10 monclient: _renew_subs 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c881382e0 con 0x7f7c8813f710 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 10 monclient: _renew_subs 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c880ab8b0 con 0x7f7c8813f710 2024-03-25T17:34:23.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.992+0000 7f7c8e4f1640 1 librados: init done 2024-03-25T17:34:23.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.993+0000 7f7c8e4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:23.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.993+0000 7f7c8e4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:23.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.993+0000 7f7c8e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:23.995 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:23.993+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88137a20 con 0x7f7c8813f710 2024-03-25T17:34:24.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c7c0011f0 con 0x7f7c8813f710 2024-03-25T17:34:24.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:24.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7c7c004c30 con 0x7f7c8813f710 2024-03-25T17:34:24.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c84ff9640 10 monclient: dump: 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:24.287 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:24.288 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:24.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.285+0000 7f7c867fc640 4 set_mon_vals no callback set 2024-03-25T17:34:24.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.286+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7c7c004e50 con 0x7f7c8813f710 2024-03-25T17:34:24.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.287+0000 7f7c87fff640 10 monclient: get_auth_request con 0x7f7c7406ddc0 auth_method 0 2024-03-25T17:34:24.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.287+0000 7f7c84ff9640 10 client.4464.objecter ms_handle_connect 0x7f7c7406ddc0 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f7c7c0549e0 con 0x7f7c8813f710 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 10 client.4464.objecter ms_dispatch 0x7f7c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c7c050320 con 0x7f7c8813f710 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:24.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.293+0000 7f7c84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:24.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.547+0000 7f7c8e4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:24.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.547+0000 7f7c8e4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:24.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.547+0000 7f7c8e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:24.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.547+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c8813d010 con 0x7f7c8813f710 2024-03-25T17:34:24.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.576+0000 7f7c84ff9640 1 -- 172.21.15.96:0/1789322369 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f7c7c052420 con 0x7f7c8813f710 2024-03-25T17:34:24.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.576+0000 7f7c84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:24.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.576+0000 7f7c84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:24.577 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:24.577 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:24.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 10 librados: watch_flush enter 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 10 librados: watch_flush exit 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 10 client.4464.objecter successfully canceled tick 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 >> v1:172.21.15.203:6800/3611964061 conn(0x7f7c7406ddc0 legacy=0x7f7c74070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 10 monclient: shutdown 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 >> v1:172.21.15.96:6789/0 conn(0x7f7c8813f710 legacy=0x7f7c88132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 shutdown_connections 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 >> 172.21.15.96:0/1789322369 conn(0x7f7c880b7ea0 msgr2=0x7f7c88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 shutdown_connections 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 -- 172.21.15.96:0/1789322369 wait complete. 2024-03-25T17:34:24.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:24.578+0000 7f7c8e4f1640 1 librados: shutdown 2024-03-25T17:34:24.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:31.398 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.510+0000 7f87fe5f4640 10 monclient: get_monmap_and_config 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.510+0000 7f87fe5f4640 10 monclient: build_initial_monmap 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.510+0000 7f87fe5f4640 10 monclient: monmap: 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.511 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:31.511975+0000 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:31.511975+0000 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:31.512 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.511+0000 7f87fe5f4640 1 Processor -- start 2024-03-25T17:34:31.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.511+0000 7f87fe5f4640 1 -- start start 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.511+0000 7f87fe5f4640 10 monclient: init 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f87f80a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f87f813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f87f81430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8142520 con 0x7f87f81430c0 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80b28a0 con 0x7f87f813f710 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80b2a30 con 0x7f87f80a98f0 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:31.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.512+0000 7f87fe5f4640 10 monclient(hunting): authenticate will time out at 5629.794790s 2024-03-25T17:34:31.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.524+0000 7f87f77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f87f813f710 0x7f87f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:59482/0 (socket says 172.21.15.96:59482) 2024-03-25T17:34:31.525 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.524+0000 7f87f77fe640 1 -- 172.21.15.96:0/745492079 learned_addr learned my addr 172.21.15.96:0/745492079 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321596795 0 0) 0x7f87f80b28a0 con 0x7f87f813f710 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 10 monclient(hunting): my global_id is 4453 2024-03-25T17:34:31.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.538+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e0003520 con 0x7f87f813f710 2024-03-25T17:34:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602667296 0 0) 0x7f87f80b2a30 con 0x7f87f80a98f0 2024-03-25T17:34:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:31.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:31.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T17:34:31.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.545+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f80b28a0 con 0x7f87f80a98f0 2024-03-25T17:34:31.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836700566 0 0) 0x7f87e0003520 con 0x7f87f813f710 2024-03-25T17:34:31.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 >> v1:172.21.15.96:6790/0 conn(0x7f87f80a98f0 legacy=0x7f87f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 >> v1:172.21.15.96:6789/0 conn(0x7f87f81430c0 legacy=0x7f87f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 monclient: found mon.noname-c 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f80b44c0 con 0x7f87f813f710 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87ec001690 con 0x7f87f813f710 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87fe5f4640 5 monclient: authenticate success, global_id 4453 2024-03-25T17:34:31.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87fe5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87f6ffd640 10 monclient: dump: 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:31.553 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.550+0000 7f87fe5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:31.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ec0017f0 con 0x7f87f813f710 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 1 -- 172.21.15.96:0/745492079 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87ec001c00 con 0x7f87f813f710 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87fe5f4640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87fe5f4640 4 set_mon_vals no callback set 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87f6ffd640 10 monclient: dump: 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:31.583 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.584 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:31.584 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:31.584 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:31.584 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87fe5f4640 10 monclient: shutdown 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87fe5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.582+0000 7f87fe5f4640 1 -- 172.21.15.96:0/745492079 >> v1:172.21.15.203:6789/0 conn(0x7f87f813f710 legacy=0x7f87f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 1 -- 172.21.15.96:0/745492079 shutdown_connections 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 1 -- 172.21.15.96:0/745492079 >> 172.21.15.96:0/745492079 conn(0x7f87f80b7ea0 msgr2=0x7f87f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 1 -- 172.21.15.96:0/745492079 shutdown_connections 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 1 -- 172.21.15.96:0/745492079 wait complete. 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 10 monclient: build_initial_monmap 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.583+0000 7f87fe5f4640 10 monclient: monmap: 2024-03-25T17:34:31.586 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 librados: starting msgr at 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 librados: starting objecter 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 Processor -- start 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 -- start start 2024-03-25T17:34:31.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 librados: setting wanted keys 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 1 librados: calling monclient init 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.584+0000 7f87fe5f4640 10 monclient: init 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f87f80a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f87f813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f87f81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f81d77a0 con 0x7f87f813f710 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8138e40 con 0x7f87f81430c0 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80aba30 con 0x7f87f80a98f0 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:31.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.585+0000 7f87fe5f4640 10 monclient(hunting): authenticate will time out at 5629.867184s 2024-03-25T17:34:31.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.587+0000 7f87fcb91640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f87f81430c0 0x7f87f8134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:59542/0 (socket says 172.21.15.96:59542) 2024-03-25T17:34:31.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.587+0000 7f87fcb91640 1 -- 172.21.15.96:0/76084323 learned_addr learned my addr 172.21.15.96:0/76084323 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:31.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.591+0000 7f87f4ff9640 10 client.?.objecter ms_handle_connect 0x7f87f81430c0 2024-03-25T17:34:31.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.591+0000 7f87f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:31.597 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.596+0000 7f87f4ff9640 10 client.?.objecter ms_handle_connect 0x7f87f80a98f0 2024-03-25T17:34:31.597 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.596+0000 7f87f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722574718 0 0) 0x7f87f8138e40 con 0x7f87f81430c0 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 10 monclient(hunting): my global_id is 4456 2024-03-25T17:34:31.602 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.601+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e4004eb0 con 0x7f87f81430c0 2024-03-25T17:34:31.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022016059 0 0) 0x7f87f80aba30 con 0x7f87f80a98f0 2024-03-25T17:34:31.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:31.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:31.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:31.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 10 monclient(hunting): my global_id is 4457 2024-03-25T17:34:31.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.607+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8138e40 con 0x7f87f80a98f0 2024-03-25T17:34:31.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.610+0000 7f87f4ff9640 10 client.?.objecter ms_handle_connect 0x7f87f813f710 2024-03-25T17:34:31.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.610+0000 7f87f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189416667 0 0) 0x7f87e4004eb0 con 0x7f87f81430c0 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87f80aba30 con 0x7f87f81430c0 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87f0001c30 con 0x7f87f81430c0 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 10 monclient(hunting): dump: 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.616 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.615+0000 7f87f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2415757974 0 0) 0x7f87f8138e40 con 0x7f87f80a98f0 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87e4004eb0 con 0x7f87f80a98f0 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87e8001930 con 0x7f87f80a98f0 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.621+0000 7f87f4ff9640 10 monclient(hunting): dump: 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:31.622 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:31.623 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 509717807 0 0) 0x7f87f80aba30 con 0x7f87f81430c0 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 >> v1:172.21.15.96:6790/0 conn(0x7f87f80a98f0 legacy=0x7f87f8138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 >> v1:172.21.15.96:6789/0 conn(0x7f87f813f710 legacy=0x7f87f81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 1 monclient: found mon.b 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f81d8340 con 0x7f87f81430c0 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 10 monclient: _finish_auth 0 2024-03-25T17:34:31.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 5 monclient: authenticate success, global_id 4456 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 10 monclient: _renew_subs 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f8134820 con 0x7f87f81430c0 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 10 monclient: _renew_subs 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f81380c0 con 0x7f87f81430c0 2024-03-25T17:34:31.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.625+0000 7f87fe5f4640 1 librados: init done 2024-03-25T17:34:31.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.626+0000 7f87fe5f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:31.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.626+0000 7f87fe5f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:31.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.626+0000 7f87fe5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:31.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.626+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f80627b0 con 0x7f87f81430c0 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87f0003990 con 0x7f87f81430c0 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87f0003da0 con 0x7f87f81430c0 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f4ff9640 10 monclient: dump: 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:31.650 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.649+0000 7f87f67fc640 4 set_mon_vals no callback set 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.650+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f87f0003fc0 con 0x7f87f81430c0 2024-03-25T17:34:31.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.650+0000 7f87f7fff640 10 monclient: get_auth_request con 0x7f87e406de90 auth_method 0 2024-03-25T17:34:31.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.650+0000 7f87f4ff9640 10 client.4456.objecter ms_handle_connect 0x7f87e406de90 2024-03-25T17:34:31.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.652+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f87f0054ab0 con 0x7f87f81430c0 2024-03-25T17:34:31.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.652+0000 7f87f4ff9640 10 client.4456.objecter ms_dispatch 0x7f87f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:31.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.652+0000 7f87f4ff9640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:31.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.652+0000 7f87f4ff9640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:31.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.653+0000 7f87f4ff9640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T17:34:31.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.921+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87f00505c0 con 0x7f87f81430c0 2024-03-25T17:34:31.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.921+0000 7f87f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:31.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:31.921+0000 7f87f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:32.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.172+0000 7f87fe5f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:32.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.172+0000 7f87fe5f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:32.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.172+0000 7f87fe5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:32.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.172+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87f814e8b0 con 0x7f87f81430c0 2024-03-25T17:34:32.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.199+0000 7f87f4ff9640 1 -- 172.21.15.96:0/76084323 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f87f00524f0 con 0x7f87f81430c0 2024-03-25T17:34:32.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.199+0000 7f87f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:32.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.199+0000 7f87f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:32.200 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:32.200 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 10 librados: watch_flush enter 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 10 librados: watch_flush exit 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 10 client.4456.objecter successfully canceled tick 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 >> v1:172.21.15.203:6800/3611964061 conn(0x7f87e406de90 legacy=0x7f87e4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 10 monclient: shutdown 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 >> v1:172.21.15.203:6789/0 conn(0x7f87f81430c0 legacy=0x7f87f8134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 shutdown_connections 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.201+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 >> 172.21.15.96:0/76084323 conn(0x7f87f80b7ea0 msgr2=0x7f87f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:32.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.202+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 shutdown_connections 2024-03-25T17:34:32.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.202+0000 7f87fe5f4640 1 -- 172.21.15.96:0/76084323 wait complete. 2024-03-25T17:34:32.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:32.202+0000 7f87fe5f4640 1 librados: shutdown 2024-03-25T17:34:32.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:39.022 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.134+0000 7f9347fc1640 10 monclient: get_monmap_and_config 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.134+0000 7f9347fc1640 10 monclient: build_initial_monmap 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.134+0000 7f9347fc1640 10 monclient: monmap: 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:39.136135+0000 2024-03-25T17:34:39.136 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:39.136135+0000 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.135+0000 7f9347fc1640 1 Processor -- start 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.135+0000 7f9347fc1640 1 -- start start 2024-03-25T17:34:39.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.135+0000 7f9347fc1640 10 monclient: init 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f93400a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f934013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f93401430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9340142520 con 0x7f934013f710 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93400b28a0 con 0x7f93401430c0 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93400b2a30 con 0x7f93400a98f0 2024-03-25T17:34:39.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:39.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.136+0000 7f9347fc1640 10 monclient(hunting): authenticate will time out at 5637.418439s 2024-03-25T17:34:39.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.145+0000 7f934655e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f93401430c0 0x7f934014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:50268/0 (socket says 172.21.15.96:50268) 2024-03-25T17:34:39.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.146+0000 7f934655e640 1 -- 172.21.15.96:0/545796223 learned_addr learned my addr 172.21.15.96:0/545796223 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:39.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845380593 0 0) 0x7f93400b28a0 con 0x7f93401430c0 2024-03-25T17:34:39.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 10 monclient(hunting): _negotiate 2024-03-25T17:34:39.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:39.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:39.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 10 monclient(hunting): my global_id is 4459 2024-03-25T17:34:39.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.152+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c003520 con 0x7f93401430c0 2024-03-25T17:34:39.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109885245 0 0) 0x7f9340142520 con 0x7f934013f710 2024-03-25T17:34:39.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 10 monclient(hunting): _negotiate 2024-03-25T17:34:39.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:39.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:39.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 10 monclient(hunting): my global_id is 4473 2024-03-25T17:34:39.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.156+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93400b28a0 con 0x7f934013f710 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 673089082 0 0) 0x7f932c003520 con 0x7f93401430c0 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 >> v1:172.21.15.96:6790/0 conn(0x7f93400a98f0 legacy=0x7f93400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 >> v1:172.21.15.96:6789/0 conn(0x7f934013f710 legacy=0x7f93401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: _finish_hunting 0 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 monclient: found mon.noname-c 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400b44c0 con 0x7f93401430c0 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: _finish_auth 0 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: _check_auth_tickets 2024-03-25T17:34:39.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9338001690 con 0x7f93401430c0 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9344d5b640 10 monclient: dump: 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9347fc1640 5 monclient: authenticate success, global_id 4459 2024-03-25T17:34:39.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.169+0000 7f9347fc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:39.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93380017f0 con 0x7f93401430c0 2024-03-25T17:34:39.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 1 -- 172.21.15.96:0/545796223 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9338001c00 con 0x7f93401430c0 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9347fc1640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9347fc1640 4 set_mon_vals no callback set 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.186+0000 7f9344d5b640 10 monclient: dump: 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.187 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:39.188 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.188 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:39.188 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:39.188 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:39.188 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 10 monclient: shutdown 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 1 -- 172.21.15.96:0/545796223 >> v1:172.21.15.203:6789/0 conn(0x7f93401430c0 legacy=0x7f934014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 1 -- 172.21.15.96:0/545796223 shutdown_connections 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 1 -- 172.21.15.96:0/545796223 >> 172.21.15.96:0/545796223 conn(0x7f93400b7ea0 msgr2=0x7f93400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 1 -- 172.21.15.96:0/545796223 shutdown_connections 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 1 -- 172.21.15.96:0/545796223 wait complete. 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.187+0000 7f9347fc1640 10 monclient: build_initial_monmap 2024-03-25T17:34:39.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 10 monclient: monmap: 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 librados: starting msgr at 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 librados: starting objecter 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 Processor -- start 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 -- start start 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 librados: setting wanted keys 2024-03-25T17:34:39.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 1 librados: calling monclient init 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.188+0000 7f9347fc1640 10 monclient: init 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f93400a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f934013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f93401430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93401d77f0 con 0x7f93401430c0 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9340139980 con 0x7f93400a98f0 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93401d51f0 con 0x7f934013f710 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:39.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.189+0000 7f9347fc1640 10 monclient(hunting): authenticate will time out at 5637.471227s 2024-03-25T17:34:39.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.201+0000 7f9345d5d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f93400a98f0 0x7f93401d7cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:50294/0 (socket says 172.21.15.96:50294) 2024-03-25T17:34:39.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.201+0000 7f9345d5d640 1 -- 172.21.15.96:0/2527892946 learned_addr learned my addr 172.21.15.96:0/2527892946 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:39.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.204+0000 7f93367fc640 10 client.?.objecter ms_handle_connect 0x7f93400a98f0 2024-03-25T17:34:39.205 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.204+0000 7f93367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:39.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.209+0000 7f93367fc640 10 client.?.objecter ms_handle_connect 0x7f934013f710 2024-03-25T17:34:39.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.209+0000 7f93367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:39.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.211+0000 7f93367fc640 10 client.?.objecter ms_handle_connect 0x7f93401430c0 2024-03-25T17:34:39.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.211+0000 7f93367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.217+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373620278 0 0) 0x7f9340139980 con 0x7f93400a98f0 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.218+0000 7f93367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.218+0000 7f93367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.218+0000 7f93367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.218+0000 7f93367fc640 10 monclient(hunting): my global_id is 4462 2024-03-25T17:34:39.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.218+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324004eb0 con 0x7f93400a98f0 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042939766 0 0) 0x7f93401d51f0 con 0x7f934013f710 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 10 monclient(hunting): my global_id is 4463 2024-03-25T17:34:39.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.222+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9340139980 con 0x7f934013f710 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312707740 0 0) 0x7f93401d77f0 con 0x7f93401430c0 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 10 monclient(hunting): my global_id is 4476 2024-03-25T17:34:39.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.234+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93401d51f0 con 0x7f93401430c0 2024-03-25T17:34:39.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36407627 0 0) 0x7f9324004eb0 con 0x7f93400a98f0 2024-03-25T17:34:39.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93401d77f0 con 0x7f93400a98f0 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f933c001db0 con 0x7f93400a98f0 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 10 monclient(hunting): dump: 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.239+0000 7f93367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3914359433 0 0) 0x7f9340139980 con 0x7f934013f710 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9324004eb0 con 0x7f934013f710 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9330001930 con 0x7f934013f710 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.250+0000 7f93367fc640 10 monclient(hunting): dump: 2024-03-25T17:34:39.251 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:39.252 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2275558830 0 0) 0x7f93401d77f0 con 0x7f93400a98f0 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 >> v1:172.21.15.96:6790/0 conn(0x7f934013f710 legacy=0x7f93401381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 >> v1:172.21.15.96:6789/0 conn(0x7f93401430c0 legacy=0x7f93401e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 10 monclient: _finish_hunting 0 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 1 monclient: found mon.b 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93401d8400 con 0x7f93400a98f0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 10 monclient: _finish_auth 0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 10 monclient: _check_auth_tickets 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f93367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 5 monclient: authenticate success, global_id 4462 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 10 monclient: _renew_subs 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93401388d0 con 0x7f93400a98f0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 10 monclient: _renew_subs 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93401e35c0 con 0x7f93400a98f0 2024-03-25T17:34:39.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.254+0000 7f9347fc1640 1 librados: init done 2024-03-25T17:34:39.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.255+0000 7f9347fc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:39.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.255+0000 7f9347fc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:39.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.255+0000 7f9347fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.255+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934014f330 con 0x7f93400a98f0 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f933c001ff0 con 0x7f93400a98f0 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f933c003c40 con 0x7f93400a98f0 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f93367fc640 10 monclient: dump: 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:39.568 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.567+0000 7f9337fff640 4 set_mon_vals no callback set 2024-03-25T17:34:39.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.568+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f933c003e20 con 0x7f93400a98f0 2024-03-25T17:34:39.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.568+0000 7f934555c640 10 monclient: get_auth_request con 0x7f932406df20 auth_method 0 2024-03-25T17:34:39.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.569+0000 7f93367fc640 10 client.4462.objecter ms_handle_connect 0x7f932406df20 2024-03-25T17:34:39.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.573+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f933c054b00 con 0x7f93400a98f0 2024-03-25T17:34:39.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.573+0000 7f93367fc640 10 client.4462.objecter ms_dispatch 0x7f9340000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.573+0000 7f93367fc640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.573+0000 7f93367fc640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.573+0000 7f93367fc640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.574+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f933c050610 con 0x7f93400a98f0 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.574+0000 7f93367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:39.575 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.574+0000 7f93367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:39.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.827+0000 7f9347fc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:39.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.827+0000 7f9347fc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:39.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.827+0000 7f9347fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:34:39.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.827+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93400627b0 con 0x7f93400a98f0 2024-03-25T17:34:39.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.865+0000 7f93367fc640 1 -- 172.21.15.96:0/2527892946 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f933c052540 con 0x7f93400a98f0 2024-03-25T17:34:39.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.865+0000 7f93367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:39.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.865+0000 7f93367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:39.866 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:39.866 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.867+0000 7f9347fc1640 10 librados: watch_flush enter 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.867+0000 7f9347fc1640 10 librados: watch_flush exit 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.867+0000 7f9347fc1640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.867+0000 7f9347fc1640 10 client.4462.objecter successfully canceled tick 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 >> v1:172.21.15.203:6800/3611964061 conn(0x7f932406df20 legacy=0x7f93240703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 10 monclient: shutdown 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 >> v1:172.21.15.203:6789/0 conn(0x7f93400a98f0 legacy=0x7f93401d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 shutdown_connections 2024-03-25T17:34:39.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 >> 172.21.15.96:0/2527892946 conn(0x7f93400b7ea0 msgr2=0x7f9340074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:39.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 shutdown_connections 2024-03-25T17:34:39.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 -- 172.21.15.96:0/2527892946 wait complete. 2024-03-25T17:34:39.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:39.868+0000 7f9347fc1640 1 librados: shutdown 2024-03-25T17:34:39.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:40.807 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:05.033 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.855 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.070 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.856 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:05.083 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.858 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.073 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.879 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:05.105 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.890 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.105 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.908 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:05.134 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.914 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:05.140 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:40.947 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:05.174 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:46.688 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:46.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 10 monclient: get_monmap_and_config 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 10 monclient: build_initial_monmap 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 10 monclient: monmap: 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:46.805465+0000 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:46.805465+0000 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:46.805 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 1 Processor -- start 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 1 -- start start 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.804+0000 7f333cbe2640 10 monclient: init 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-a con 0x7f33380a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-b con 0x7f333813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-c con 0x7f33381430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.805+0000 7f333cbe2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3338142520 con 0x7f33380a98f0 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.806+0000 7f333cbe2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33380b28a0 con 0x7f33381430c0 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.806+0000 7f333cbe2640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33380b2a30 con 0x7f333813f710 2024-03-25T17:34:46.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.806+0000 7f333cbe2640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:46.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.806+0000 7f333cbe2640 10 monclient(hunting): authenticate will time out at 5645.088009s 2024-03-25T17:34:46.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.819+0000 7f3335d9b640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f333813f710 0x7f33381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:35516/0 (socket says 172.21.15.96:35516) 2024-03-25T17:34:46.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.819+0000 7f3335d9b640 1 -- 172.21.15.96:0/3880275266 learned_addr learned my addr 172.21.15.96:0/3880275266 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350706996 0 0) 0x7f33380b2a30 con 0x7f333813f710 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 10 monclient(hunting): _negotiate 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 10 monclient(hunting): my global_id is 4466 2024-03-25T17:34:46.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.837+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3320003520 con 0x7f333813f710 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434411266 0 0) 0x7f3338142520 con 0x7f33380a98f0 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 10 monclient(hunting): _negotiate 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 10 monclient(hunting): my global_id is 4479 2024-03-25T17:34:46.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.901+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33380b2a30 con 0x7f33380a98f0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894523399 0 0) 0x7f33380b2a30 con 0x7f33380a98f0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 >> v1:172.21.15.96:6790/0 conn(0x7f333813f710 legacy=0x7f33381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 >> v1:172.21.15.203:6789/0 conn(0x7f33381430c0 legacy=0x7f333814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: _finish_hunting 0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 monclient: found mon.noname-a 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33380b44c0 con 0x7f33380a98f0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: _finish_auth 0 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: _check_auth_tickets 2024-03-25T17:34:46.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3328001670 con 0x7f33380a98f0 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.941+0000 7f333559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.942+0000 7f333559a640 10 monclient: dump: 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.942+0000 7f333cbe2640 5 monclient: authenticate success, global_id 4479 2024-03-25T17:34:46.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.942+0000 7f333cbe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:46.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33280017d0 con 0x7f33380a98f0 2024-03-25T17:34:46.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:46.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 1 -- 172.21.15.96:0/3880275266 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3328001ba0 con 0x7f33380a98f0 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333559a640 10 monclient: dump: 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333cbe2640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:46.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.961+0000 7f333cbe2640 4 set_mon_vals no callback set 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 10 monclient: shutdown 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 1 -- 172.21.15.96:0/3880275266 >> v1:172.21.15.96:6789/0 conn(0x7f33380a98f0 legacy=0x7f33380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 1 -- 172.21.15.96:0/3880275266 shutdown_connections 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 1 -- 172.21.15.96:0/3880275266 >> 172.21.15.96:0/3880275266 conn(0x7f33380b7ea0 msgr2=0x7f33380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 1 -- 172.21.15.96:0/3880275266 shutdown_connections 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.962+0000 7f333cbe2640 1 -- 172.21.15.96:0/3880275266 wait complete. 2024-03-25T17:34:46.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.963+0000 7f333cbe2640 10 monclient: build_initial_monmap 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.963+0000 7f333cbe2640 10 monclient: monmap: 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 librados: starting msgr at 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 librados: starting objecter 2024-03-25T17:34:46.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 Processor -- start 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 -- start start 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 librados: setting wanted keys 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 1 librados: calling monclient init 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.964+0000 7f333cbe2640 10 monclient: init 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-b con 0x7f33380a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:46.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-a con 0x7f333813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient(hunting): picked mon.noname-c con 0x7f33381430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33381daa20 con 0x7f333813f710 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33381371e0 con 0x7f33380a98f0 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33381374c0 con 0x7f33381430c0 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:46.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.965+0000 7f333cbe2640 10 monclient(hunting): authenticate will time out at 5645.247141s 2024-03-25T17:34:46.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.967+0000 7f333659c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f33380a98f0 0x7f33381352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:53996/0 (socket says 172.21.15.96:53996) 2024-03-25T17:34:46.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.967+0000 7f333659c640 1 -- 172.21.15.96:0/2705163067 learned_addr learned my addr 172.21.15.96:0/2705163067 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:46.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.968+0000 7f3326ffd640 10 client.?.objecter ms_handle_connect 0x7f33380a98f0 2024-03-25T17:34:46.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.968+0000 7f3326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:46.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.980+0000 7f3326ffd640 10 client.?.objecter ms_handle_connect 0x7f333813f710 2024-03-25T17:34:46.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.980+0000 7f3326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:46.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.981+0000 7f3326ffd640 10 client.?.objecter ms_handle_connect 0x7f33381430c0 2024-03-25T17:34:46.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:46.981+0000 7f3326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231728054 0 0) 0x7f33381daa20 con 0x7f333813f710 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 10 monclient(hunting): my global_id is 4482 2024-03-25T17:34:47.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.044+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3318004eb0 con 0x7f333813f710 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007516360 0 0) 0x7f33381374c0 con 0x7f33381430c0 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T17:34:47.051 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.050+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33381daa20 con 0x7f33381430c0 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330323818 0 0) 0x7f3318004eb0 con 0x7f333813f710 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33381374c0 con 0x7f333813f710 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f332c001950 con 0x7f333813f710 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 10 monclient(hunting): dump: 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:47.065 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:47.066 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:47.066 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:47.066 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:47.066 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:47.066 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.064+0000 7f3326ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826660555 0 0) 0x7f33381daa20 con 0x7f33381430c0 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3318004eb0 con 0x7f33381430c0 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3330001b00 con 0x7f33381430c0 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.075+0000 7f3326ffd640 10 monclient(hunting): dump: 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:47.076 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:47.077 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3844755808 0 0) 0x7f33381374c0 con 0x7f333813f710 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 >> v1:172.21.15.96:6790/0 conn(0x7f33381430c0 legacy=0x7f33381e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 >> v1:172.21.15.203:6789/0 conn(0x7f33380a98f0 legacy=0x7f33381352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 1 monclient: found mon.a 2024-03-25T17:34:47.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333813c040 con 0x7f333813f710 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 10 monclient: _finish_auth 0 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f3326ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 5 monclient: authenticate success, global_id 4482 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 10 monclient: _renew_subs 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33381359f0 con 0x7f333813f710 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 10 monclient: _renew_subs 2024-03-25T17:34:47.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:47.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33381d75d0 con 0x7f333813f710 2024-03-25T17:34:47.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.084+0000 7f333cbe2640 1 librados: init done 2024-03-25T17:34:47.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.085+0000 7f333cbe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:47.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.085+0000 7f333cbe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:47.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.085+0000 7f333cbe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:47.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.085+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33380627b0 con 0x7f333813f710 2024-03-25T17:34:47.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.297+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f332c0047e0 con 0x7f333813f710 2024-03-25T17:34:47.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.297+0000 7f3326ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:47.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.297+0000 7f3334d99640 4 set_mon_vals no callback set 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.372+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f332c004af0 con 0x7f333813f710 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.372+0000 7f3326ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.372+0000 7f3326ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.372+0000 7f3326ffd640 10 monclient: dump: 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:47.373 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:47.374 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:47.374 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:47.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.373+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f332c01aac0 con 0x7f333813f710 2024-03-25T17:34:47.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.373+0000 7f333659c640 10 monclient: get_auth_request con 0x7f331806de90 auth_method 0 2024-03-25T17:34:47.374 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.374+0000 7f3326ffd640 10 client.4482.objecter ms_handle_connect 0x7f331806de90 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f332c054830 con 0x7f333813f710 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 10 client.4482.objecter ms_dispatch 0x7f3338000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f332c050170 con 0x7f333813f710 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:47.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.379+0000 7f3326ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:47.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.639+0000 7f333cbe2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:47.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.639+0000 7f333cbe2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:47.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.639+0000 7f333cbe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:47.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.639+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33381dabb0 con 0x7f333813f710 2024-03-25T17:34:47.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.675+0000 7f3326ffd640 1 -- 172.21.15.96:0/2705163067 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f332c052270 con 0x7f333813f710 2024-03-25T17:34:47.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.675+0000 7f3326ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:47.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.675+0000 7f3326ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:47.676 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:47.676 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 10 librados: watch_flush enter 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 10 librados: watch_flush exit 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 10 client.4482.objecter successfully canceled tick 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 >> v1:172.21.15.203:6800/3611964061 conn(0x7f331806de90 legacy=0x7f3318070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 10 monclient: shutdown 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 >> v1:172.21.15.96:6789/0 conn(0x7f333813f710 legacy=0x7f33381daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 shutdown_connections 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 >> 172.21.15.96:0/2705163067 conn(0x7f33380b7ea0 msgr2=0x7f3338074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 shutdown_connections 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 -- 172.21.15.96:0/2705163067 wait complete. 2024-03-25T17:34:47.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:47.677+0000 7f333cbe2640 1 librados: shutdown 2024-03-25T17:34:47.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:54.495 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:54.610 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.611+0000 7ffbf304d640 10 monclient: get_monmap_and_config 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.611+0000 7ffbf304d640 10 monclient: build_initial_monmap 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.611+0000 7ffbf304d640 10 monclient: monmap: 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:34:54.611466+0000 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:34:54.611466+0000 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:34:54.611 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 1 Processor -- start 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 1 -- start start 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient: init 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbec0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbec14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.612+0000 7ffbf304d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec0af940 con 0x7ffbec14be90 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.613+0000 7ffbf304d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec0b0a70 con 0x7ffbec0b1730 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.613+0000 7ffbf304d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec0b0d50 con 0x7ffbec0b4d60 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.613+0000 7ffbf304d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:54.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.613+0000 7ffbf304d640 10 monclient(hunting): authenticate will time out at 5652.893966s 2024-03-25T17:34:54.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.615+0000 7ffbf0de9640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ffbec0b4d60 0x7ffbec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:35016/0 (socket says 172.21.15.96:35016) 2024-03-25T17:34:54.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.615+0000 7ffbf0de9640 1 -- 172.21.15.96:0/3041011571 learned_addr learned my addr 172.21.15.96:0/3041011571 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:54.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664179180 0 0) 0x7ffbec0b0d50 con 0x7ffbec0b4d60 2024-03-25T17:34:54.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:54.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:54.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:54.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T17:34:54.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.625+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbdc003520 con 0x7ffbec0b4d60 2024-03-25T17:34:54.630 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318152888 0 0) 0x7ffbec0af940 con 0x7ffbec14be90 2024-03-25T17:34:54.630 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:54.630 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:54.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:54.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T17:34:54.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.631+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbec0b0d50 con 0x7ffbec14be90 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 796863258 0 0) 0x7ffbec0b0d50 con 0x7ffbec14be90 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 >> v1:172.21.15.96:6790/0 conn(0x7ffbec0b4d60 legacy=0x7ffbec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 >> v1:172.21.15.203:6789/0 conn(0x7ffbec0b1730 legacy=0x7ffbec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 monclient: found mon.noname-a 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec0b1030 con 0x7ffbec14be90 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:54.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffbe40019f0 con 0x7ffbec14be90 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbf304d640 5 monclient: authenticate success, global_id 4485 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbf304d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbeb7fe640 10 monclient: dump: 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:54.636 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:54.637 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.635+0000 7ffbf304d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbe4001b50 con 0x7ffbec14be90 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 1 -- 172.21.15.96:0/3041011571 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffbe4001f20 con 0x7ffbec14be90 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbf304d640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbf304d640 4 set_mon_vals no callback set 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:54.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.645+0000 7ffbeb7fe640 10 monclient: dump: 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:54.646 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 10 monclient: shutdown 2024-03-25T17:34:54.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:54.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 1 -- 172.21.15.96:0/3041011571 >> v1:172.21.15.96:6789/0 conn(0x7ffbec14be90 legacy=0x7ffbec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 1 -- 172.21.15.96:0/3041011571 shutdown_connections 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 1 -- 172.21.15.96:0/3041011571 >> 172.21.15.96:0/3041011571 conn(0x7ffbec0617a0 msgr2=0x7ffbec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 1 -- 172.21.15.96:0/3041011571 shutdown_connections 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.646+0000 7ffbf304d640 1 -- 172.21.15.96:0/3041011571 wait complete. 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 10 monclient: build_initial_monmap 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 10 monclient: monmap: 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:34:54.649 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 librados: starting msgr at 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 librados: starting objecter 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 Processor -- start 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 -- start start 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 librados: setting wanted keys 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 1 librados: calling monclient init 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.647+0000 7ffbf304d640 10 monclient: init 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbec0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:34:54.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbec14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:34:54.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec1d7690 con 0x7ffbec0b4d60 2024-03-25T17:34:54.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec142ff0 con 0x7ffbec14be90 2024-03-25T17:34:54.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec143900 con 0x7ffbec0b1730 2024-03-25T17:34:54.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:54.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.648+0000 7ffbf304d640 10 monclient(hunting): authenticate will time out at 5652.929869s 2024-03-25T17:34:54.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.662+0000 7ffbf0de9640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ffbec0b1730 0x7ffbec141510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:35058/0 (socket says 172.21.15.96:35058) 2024-03-25T17:34:54.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.662+0000 7ffbf0de9640 1 -- 172.21.15.96:0/3516425442 learned_addr learned my addr 172.21.15.96:0/3516425442 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:34:54.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.664+0000 7ffbe97fa640 10 client.?.objecter ms_handle_connect 0x7ffbec0b1730 2024-03-25T17:34:54.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.664+0000 7ffbe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603133013 0 0) 0x7ffbec143900 con 0x7ffbec0b1730 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T17:34:54.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd0004eb0 con 0x7ffbec0b1730 2024-03-25T17:34:54.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 client.?.objecter ms_handle_connect 0x7ffbec14be90 2024-03-25T17:34:54.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.668+0000 7ffbe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:54.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.670+0000 7ffbe97fa640 10 client.?.objecter ms_handle_connect 0x7ffbec0b4d60 2024-03-25T17:34:54.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.670+0000 7ffbe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:54.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936053475 0 0) 0x7ffbec1d7690 con 0x7ffbec0b4d60 2024-03-25T17:34:54.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:54.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:54.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:54.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 10 monclient(hunting): my global_id is 4488 2024-03-25T17:34:54.675 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.675+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbec143900 con 0x7ffbec0b4d60 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292087640 0 0) 0x7ffbec142ff0 con 0x7ffbec14be90 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 10 monclient(hunting): my global_id is 4474 2024-03-25T17:34:54.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.681+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbec1d7690 con 0x7ffbec14be90 2024-03-25T17:34:54.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593510174 0 0) 0x7ffbec143900 con 0x7ffbec0b4d60 2024-03-25T17:34:54.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbec142ff0 con 0x7ffbec0b4d60 2024-03-25T17:34:54.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 808968789 0 0) 0x7ffbd0004eb0 con 0x7ffbec0b1730 2024-03-25T17:34:54.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbec143900 con 0x7ffbec0b1730 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffbe0001950 con 0x7ffbec0b1730 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 10 monclient(hunting): dump: 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.682+0000 7ffbe97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.689+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffbd8001950 con 0x7ffbec0b4d60 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.689+0000 7ffbe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.689+0000 7ffbe97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.689+0000 7ffbe97fa640 10 monclient(hunting): dump: 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:54.689 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 374029190 0 0) 0x7ffbec142ff0 con 0x7ffbec0b4d60 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 >> v1:172.21.15.96:6790/0 conn(0x7ffbec0b1730 legacy=0x7ffbec141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 >> v1:172.21.15.203:6789/0 conn(0x7ffbec14be90 legacy=0x7ffbec1e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 1 monclient: found mon.a 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec140990 con 0x7ffbec0b4d60 2024-03-25T17:34:54.694 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 10 monclient: _finish_auth 0 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 5 monclient: authenticate success, global_id 4488 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 10 monclient: _renew_subs 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec141c20 con 0x7ffbec0b4d60 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 10 monclient: _renew_subs 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec1d8480 con 0x7ffbec0b4d60 2024-03-25T17:34:54.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.694+0000 7ffbf304d640 1 librados: init done 2024-03-25T17:34:54.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.695+0000 7ffbf304d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:54.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.695+0000 7ffbf304d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:54.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.695+0000 7ffbf304d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:54.696 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:54.695+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec141f20 con 0x7ffbec0b4d60 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd80035f0 con 0x7ffbec0b4d60 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffbd8003a00 con 0x7ffbec0b4d60 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbe97fa640 10 monclient: dump: 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:55.007 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.007+0000 7ffbeaffd640 4 set_mon_vals no callback set 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.008+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ffbd8003c20 con 0x7ffbec0b4d60 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.008+0000 7ffbf0de9640 10 monclient: get_auth_request con 0x7ffbd006dfb0 auth_method 0 2024-03-25T17:34:55.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.009+0000 7ffbe97fa640 10 client.4488.objecter ms_handle_connect 0x7ffbd006dfb0 2024-03-25T17:34:55.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ffbd80535d0 con 0x7ffbec0b4d60 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 10 client.4488.objecter ms_dispatch 0x7ffbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.014+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbd804f0e0 con 0x7ffbec0b4d60 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.015+0000 7ffbe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:55.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.015+0000 7ffbe97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:55.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.231+0000 7ffbf304d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:55.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.231+0000 7ffbf304d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:55.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.231+0000 7ffbf304d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:34:55.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.231+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbec0627b0 con 0x7ffbec0b4d60 2024-03-25T17:34:55.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.306+0000 7ffbe97fa640 1 -- 172.21.15.96:0/3516425442 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ffbd8051010 con 0x7ffbec0b4d60 2024-03-25T17:34:55.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.306+0000 7ffbe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:55.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.306+0000 7ffbe97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:55.306 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:34:55.307 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:55.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 10 librados: watch_flush enter 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 10 librados: watch_flush exit 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 10 client.4488.objecter successfully canceled tick 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 >> v1:172.21.15.203:6800/3611964061 conn(0x7ffbd006dfb0 legacy=0x7ffbd0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 10 monclient: shutdown 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 >> v1:172.21.15.96:6789/0 conn(0x7ffbec0b4d60 legacy=0x7ffbec1d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 shutdown_connections 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.308+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 >> 172.21.15.96:0/3516425442 conn(0x7ffbec0617a0 msgr2=0x7ffbec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.309+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 shutdown_connections 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.309+0000 7ffbf304d640 1 -- 172.21.15.96:0/3516425442 wait complete. 2024-03-25T17:34:55.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:34:55.309+0000 7ffbf304d640 1 librados: shutdown 2024-03-25T17:34:55.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:00.898 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:25.124 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.906 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:25.132 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.942 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:25.169 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.172 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.966 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.181 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.970 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:25.197 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.980 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:25.207 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:00.997 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.212 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:01.005 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:25.232 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:02.127 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.245+0000 7fe866014640 10 monclient: get_monmap_and_config 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.245+0000 7fe866014640 10 monclient: build_initial_monmap 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.245+0000 7fe866014640 10 monclient: monmap: 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:02.245482+0000 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:02.245482+0000 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:02.245 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:02.246 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:02.246 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.246+0000 7fe866014640 1 Processor -- start 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.246+0000 7fe866014640 1 -- start start 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.246+0000 7fe866014640 10 monclient: init 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient(hunting): picked mon.noname-b con 0x7fe8600a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient(hunting): picked mon.noname-c con 0x7fe86013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient(hunting): picked mon.noname-a con 0x7fe8601430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:02.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe860142520 con 0x7fe8601430c0 2024-03-25T17:35:02.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600b28a0 con 0x7fe86013f710 2024-03-25T17:35:02.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600b2a30 con 0x7fe8600a98f0 2024-03-25T17:35:02.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:02.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.247+0000 7fe866014640 10 monclient(hunting): authenticate will time out at 5660.528349s 2024-03-25T17:35:02.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.259+0000 7fe85f7fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fe8600a98f0 0x7fe8600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:39932/0 (socket says 172.21.15.96:39932) 2024-03-25T17:35:02.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.259+0000 7fe85f7fe640 1 -- 172.21.15.96:0/4239587693 learned_addr learned my addr 172.21.15.96:0/4239587693 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019088990 0 0) 0x7fe8600b2a30 con 0x7fe8600a98f0 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 10 monclient(hunting): my global_id is 4478 2024-03-25T17:35:02.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.266+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe848003520 con 0x7fe8600a98f0 2024-03-25T17:35:02.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267327480 0 0) 0x7fe8600b28a0 con 0x7fe86013f710 2024-03-25T17:35:02.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:35:02.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:02.279 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:02.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 10 monclient(hunting): my global_id is 4477 2024-03-25T17:35:02.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.280+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8600b2a30 con 0x7fe86013f710 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043715266 0 0) 0x7fe848003520 con 0x7fe8600a98f0 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 >> v1:172.21.15.203:6789/0 conn(0x7fe86013f710 legacy=0x7fe8601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 >> v1:172.21.15.96:6789/0 conn(0x7fe8601430c0 legacy=0x7fe86014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 monclient: found mon.noname-b 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8600b44c0 con 0x7fe8600a98f0 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:35:02.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe850001690 con 0x7fe8600a98f0 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe85e7fc640 10 monclient: dump: 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe866014640 5 monclient: authenticate success, global_id 4478 2024-03-25T17:35:02.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.292+0000 7fe866014640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:02.299 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe850001870 con 0x7fe8600a98f0 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 1 -- 172.21.15.96:0/4239587693 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe850001c80 con 0x7fe8600a98f0 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe85e7fc640 10 monclient: dump: 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe866014640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:02.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.299+0000 7fe866014640 4 set_mon_vals no callback set 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.300+0000 7fe866014640 10 monclient: shutdown 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.300+0000 7fe866014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.300+0000 7fe866014640 1 -- 172.21.15.96:0/4239587693 >> v1:172.21.15.96:6790/0 conn(0x7fe8600a98f0 legacy=0x7fe8600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.300+0000 7fe866014640 1 -- 172.21.15.96:0/4239587693 shutdown_connections 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.300+0000 7fe866014640 1 -- 172.21.15.96:0/4239587693 >> 172.21.15.96:0/4239587693 conn(0x7fe8600b7ea0 msgr2=0x7fe8600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.301+0000 7fe866014640 1 -- 172.21.15.96:0/4239587693 shutdown_connections 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.301+0000 7fe866014640 1 -- 172.21.15.96:0/4239587693 wait complete. 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.301+0000 7fe866014640 10 monclient: build_initial_monmap 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.301+0000 7fe866014640 10 monclient: monmap: 2024-03-25T17:35:02.302 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 librados: starting msgr at 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 librados: starting objecter 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 Processor -- start 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 -- start start 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 librados: setting wanted keys 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 1 librados: calling monclient init 2024-03-25T17:35:02.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.302+0000 7fe866014640 10 monclient: init 2024-03-25T17:35:02.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:02.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient(hunting): picked mon.noname-a con 0x7fe8600a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient(hunting): picked mon.noname-b con 0x7fe86013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient(hunting): picked mon.noname-c con 0x7fe8601430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8601374f0 con 0x7fe8600a98f0 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8601db360 con 0x7fe86013f710 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe860133a40 con 0x7fe8601430c0 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:02.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.303+0000 7fe866014640 10 monclient(hunting): authenticate will time out at 5660.584241s 2024-03-25T17:35:02.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.315+0000 7fe85f7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe8600a98f0 0x7fe8601dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44236/0 (socket says 172.21.15.96:44236) 2024-03-25T17:35:02.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.315+0000 7fe85f7fe640 1 -- 172.21.15.96:0/3026411732 learned_addr learned my addr 172.21.15.96:0/3026411732 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:02.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.315+0000 7fe847fff640 10 client.?.objecter ms_handle_connect 0x7fe8600a98f0 2024-03-25T17:35:02.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.315+0000 7fe847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:02.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.317+0000 7fe847fff640 10 client.?.objecter ms_handle_connect 0x7fe8601430c0 2024-03-25T17:35:02.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.317+0000 7fe847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635220091 0 0) 0x7fe8601374f0 con 0x7fe8600a98f0 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 10 monclient(hunting): my global_id is 4494 2024-03-25T17:35:02.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.320+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c004eb0 con 0x7fe8600a98f0 2024-03-25T17:35:02.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532683886 0 0) 0x7fe860133a40 con 0x7fe8601430c0 2024-03-25T17:35:02.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:02.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:02.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:02.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 10 monclient(hunting): my global_id is 4481 2024-03-25T17:35:02.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.324+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8601374f0 con 0x7fe8601430c0 2024-03-25T17:35:02.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.325+0000 7fe847fff640 10 client.?.objecter ms_handle_connect 0x7fe86013f710 2024-03-25T17:35:02.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.325+0000 7fe847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2564424374 0 0) 0x7fe84c004eb0 con 0x7fe8600a98f0 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe860133a40 con 0x7fe8600a98f0 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe850001c40 con 0x7fe8600a98f0 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 10 monclient(hunting): dump: 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:02.332 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.332+0000 7fe847fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467828239 0 0) 0x7fe8601db360 con 0x7fe86013f710 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.339+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c004eb0 con 0x7fe86013f710 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3565776348 0 0) 0x7fe860133a40 con 0x7fe8600a98f0 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 >> v1:172.21.15.96:6790/0 conn(0x7fe8601430c0 legacy=0x7fe860133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 >> v1:172.21.15.203:6789/0 conn(0x7fe86013f710 legacy=0x7fe860132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 10 monclient: _finish_hunting 0 2024-03-25T17:35:02.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 1 monclient: found mon.a 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860138000 con 0x7fe8600a98f0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 10 monclient: _finish_auth 0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 10 monclient: _check_auth_tickets 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 5 monclient: authenticate success, global_id 4494 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 10 monclient: _renew_subs 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe860136030 con 0x7fe8600a98f0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 10 monclient: _renew_subs 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:02.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8600b7570 con 0x7fe8600a98f0 2024-03-25T17:35:02.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.340+0000 7fe866014640 1 librados: init done 2024-03-25T17:35:02.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.341+0000 7fe866014640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:02.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.341+0000 7fe866014640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:02.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.341+0000 7fe866014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:02.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.341+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8600627b0 con 0x7fe8600a98f0 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe850001170 con 0x7fe8600a98f0 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe850004d90 con 0x7fe8600a98f0 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe847fff640 10 monclient: dump: 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:02.637 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:02.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.637+0000 7fe85dffb640 4 set_mon_vals no callback set 2024-03-25T17:35:02.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.638+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fe85001ada0 con 0x7fe8600a98f0 2024-03-25T17:35:02.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.638+0000 7fe85effd640 10 monclient: get_auth_request con 0x7fe84c06ddd0 auth_method 0 2024-03-25T17:35:02.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.639+0000 7fe847fff640 10 client.4494.objecter ms_handle_connect 0x7fe84c06ddd0 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fe850054a70 con 0x7fe8600a98f0 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 10 client.4494.objecter ms_dispatch 0x7fe860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8500503b0 con 0x7fe8600a98f0 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:02.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.644+0000 7fe847fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:02.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.878+0000 7fe866014640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:02.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.878+0000 7fe866014640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:02.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.878+0000 7fe866014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:02.878 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.878+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe86013e210 con 0x7fe8600a98f0 2024-03-25T17:35:02.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.923+0000 7fe847fff640 1 -- 172.21.15.96:0/3026411732 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fe8500524b0 con 0x7fe8600a98f0 2024-03-25T17:35:02.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.923+0000 7fe847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:02.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.923+0000 7fe847fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:02.923 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:02.923 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:02.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 10 librados: watch_flush enter 2024-03-25T17:35:02.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 10 librados: watch_flush exit 2024-03-25T17:35:02.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T17:35:02.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 10 client.4494.objecter successfully canceled tick 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 >> v1:172.21.15.203:6800/3611964061 conn(0x7fe84c06ddd0 legacy=0x7fe84c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 10 monclient: shutdown 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 >> v1:172.21.15.96:6789/0 conn(0x7fe8600a98f0 legacy=0x7fe8601dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 shutdown_connections 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 >> 172.21.15.96:0/3026411732 conn(0x7fe8600b7ea0 msgr2=0x7fe86013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 shutdown_connections 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 -- 172.21.15.96:0/3026411732 wait complete. 2024-03-25T17:35:02.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:02.926+0000 7fe866014640 1 librados: shutdown 2024-03-25T17:35:02.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:09.746 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:09.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.860+0000 7f3b2eac8640 10 monclient: get_monmap_and_config 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.860+0000 7f3b2eac8640 10 monclient: build_initial_monmap 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.860+0000 7f3b2eac8640 10 monclient: monmap: 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:09.861466+0000 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:09.861466+0000 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:09.861 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:09.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.860+0000 7f3b2eac8640 1 Processor -- start 2024-03-25T17:35:09.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.861+0000 7f3b2eac8640 1 -- start start 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.861+0000 7f3b2eac8640 10 monclient: init 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.861+0000 7f3b2eac8640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.861+0000 7f3b2eac8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.861+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-c con 0x7f3b280b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b280b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2814be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280af940 con 0x7f3b2814be90 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280b0a70 con 0x7f3b280b4d60 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280b0d50 con 0x7f3b280b1730 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:09.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.862+0000 7f3b2eac8640 10 monclient(hunting): authenticate will time out at 5668.144172s 2024-03-25T17:35:09.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.881+0000 7f3b27fff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f3b280b1730 0x7f3b280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:42202/0 (socket says 172.21.15.96:42202) 2024-03-25T17:35:09.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.881+0000 7f3b27fff640 1 -- 172.21.15.96:0/3310859661 learned_addr learned my addr 172.21.15.96:0/3310859661 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:09.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479107604 0 0) 0x7f3b280b0d50 con 0x7f3b280b1730 2024-03-25T17:35:09.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T17:35:09.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.893+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003520 con 0x7f3b280b1730 2024-03-25T17:35:09.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704999886 0 0) 0x7f3b280b0a70 con 0x7f3b280b4d60 2024-03-25T17:35:09.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T17:35:09.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.898+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b280b0d50 con 0x7f3b280b4d60 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221694694 0 0) 0x7f3b280af940 con 0x7f3b2814be90 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 10 monclient(hunting): my global_id is 4497 2024-03-25T17:35:09.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.912+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b280b0a70 con 0x7f3b2814be90 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201864406 0 0) 0x7f3b18003520 con 0x7f3b280b1730 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 >> v1:172.21.15.203:6789/0 conn(0x7f3b280b4d60 legacy=0x7f3b2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 >> v1:172.21.15.96:6789/0 conn(0x7f3b2814be90 legacy=0x7f3b2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 monclient: found mon.noname-b 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280b1030 con 0x7f3b280b1730 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3b14001690 con 0x7f3b280b1730 2024-03-25T17:35:09.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b277fe640 10 monclient: dump: 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b2eac8640 5 monclient: authenticate success, global_id 4484 2024-03-25T17:35:09.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.915+0000 7f3b2eac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:09.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b140017f0 con 0x7f3b280b1730 2024-03-25T17:35:09.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:09.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 1 -- 172.21.15.96:0/3310859661 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3b14001c00 con 0x7f3b280b1730 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b2eac8640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b2eac8640 4 set_mon_vals no callback set 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b277fe640 10 monclient: dump: 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:09.926 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b2eac8640 10 monclient: shutdown 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b2eac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.925+0000 7f3b2eac8640 1 -- 172.21.15.96:0/3310859661 >> v1:172.21.15.96:6790/0 conn(0x7f3b280b1730 legacy=0x7f3b280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 1 -- 172.21.15.96:0/3310859661 shutdown_connections 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 1 -- 172.21.15.96:0/3310859661 >> 172.21.15.96:0/3310859661 conn(0x7f3b280617a0 msgr2=0x7f3b28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:09.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 1 -- 172.21.15.96:0/3310859661 shutdown_connections 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 1 -- 172.21.15.96:0/3310859661 wait complete. 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 10 monclient: build_initial_monmap 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.926+0000 7f3b2eac8640 10 monclient: monmap: 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:09.929 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 librados: starting msgr at 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 librados: starting objecter 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 Processor -- start 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 -- start start 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 librados: setting wanted keys 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 1 librados: calling monclient init 2024-03-25T17:35:09.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.927+0000 7f3b2eac8640 10 monclient: init 2024-03-25T17:35:09.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:09.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:09.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-a con 0x7f3b280b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-c con 0x7f3b280b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2814be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b281404d0 con 0x7f3b280b1730 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b281daeb0 con 0x7f3b2814be90 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b281380e0 con 0x7f3b280b4d60 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:09.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.928+0000 7f3b2eac8640 10 monclient(hunting): authenticate will time out at 5668.210678s 2024-03-25T17:35:09.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.937+0000 7f3b2d065640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3b2814be90 0x7f3b28130de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:56284/0 (socket says 172.21.15.96:56284) 2024-03-25T17:35:09.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.937+0000 7f3b2d065640 1 -- 172.21.15.96:0/1343524870 learned_addr learned my addr 172.21.15.96:0/1343524870 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:09.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.939+0000 7f3b257fa640 10 client.?.objecter ms_handle_connect 0x7f3b2814be90 2024-03-25T17:35:09.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.939+0000 7f3b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:09.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.939+0000 7f3b257fa640 10 client.?.objecter ms_handle_connect 0x7f3b280b4d60 2024-03-25T17:35:09.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.939+0000 7f3b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:09.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564377433 0 0) 0x7f3b281daeb0 con 0x7f3b2814be90 2024-03-25T17:35:09.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 10 monclient(hunting): my global_id is 4486 2024-03-25T17:35:09.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.942+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b0c004eb0 con 0x7f3b2814be90 2024-03-25T17:35:09.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.947+0000 7f3b257fa640 10 client.?.objecter ms_handle_connect 0x7f3b280b1730 2024-03-25T17:35:09.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.947+0000 7f3b257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497371927 0 0) 0x7f3b0c004eb0 con 0x7f3b2814be90 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b281daeb0 con 0x7f3b2814be90 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3b20001c70 con 0x7f3b2814be90 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 10 monclient(hunting): dump: 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:09.952 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:09.953 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:09.953 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:09.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.951+0000 7f3b257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:09.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322793283 0 0) 0x7f3b281404d0 con 0x7f3b280b1730 2024-03-25T17:35:09.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 10 monclient(hunting): my global_id is 4500 2024-03-25T17:35:09.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.955+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b0c004eb0 con 0x7f3b280b1730 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937460737 0 0) 0x7f3b281380e0 con 0x7f3b280b4d60 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 10 monclient(hunting): my global_id is 4487 2024-03-25T17:35:09.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.956+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b281404d0 con 0x7f3b280b4d60 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1829716289 0 0) 0x7f3b281daeb0 con 0x7f3b2814be90 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 >> v1:172.21.15.96:6790/0 conn(0x7f3b280b4d60 legacy=0x7f3b281db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 >> v1:172.21.15.96:6789/0 conn(0x7f3b280b1730 legacy=0x7f3b281379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 1 monclient: found mon.b 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b281dbae0 con 0x7f3b2814be90 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:09.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 5 monclient: authenticate success, global_id 4486 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 10 monclient: _renew_subs 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28137310 con 0x7f3b2814be90 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 10 monclient: _renew_subs 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b28143d40 con 0x7f3b2814be90 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.967+0000 7f3b2eac8640 1 librados: init done 2024-03-25T17:35:09.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.968+0000 7f3b2eac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:09.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.968+0000 7f3b2eac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:09.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.968+0000 7f3b2eac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:09.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.968+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b281db040 con 0x7f3b2814be90 2024-03-25T17:35:09.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.971+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b20003970 con 0x7f3b2814be90 2024-03-25T17:35:09.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.971+0000 7f3b257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:09.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:09.971+0000 7f3b26ffd640 4 set_mon_vals no callback set 2024-03-25T17:35:10.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.250+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3b20003dc0 con 0x7f3b2814be90 2024-03-25T17:35:10.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.250+0000 7f3b257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:10.251 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.250+0000 7f3b257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.250+0000 7f3b257fa640 10 monclient: dump: 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:10.252 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:10.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.251+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f3b20003fa0 con 0x7f3b2814be90 2024-03-25T17:35:10.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.252+0000 7f3b2c864640 10 monclient: get_auth_request con 0x7f3b0c06de80 auth_method 0 2024-03-25T17:35:10.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.252+0000 7f3b257fa640 10 client.4486.objecter ms_handle_connect 0x7f3b0c06de80 2024-03-25T17:35:10.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.256+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f3b20054ac0 con 0x7f3b2814be90 2024-03-25T17:35:10.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.256+0000 7f3b257fa640 10 client.4486.objecter ms_dispatch 0x7f3b28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:10.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.256+0000 7f3b257fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:10.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.256+0000 7f3b257fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:10.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.256+0000 7f3b257fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T17:35:10.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.259+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b200505d0 con 0x7f3b2814be90 2024-03-25T17:35:10.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.259+0000 7f3b257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:10.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.259+0000 7f3b257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:10.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.504+0000 7f3b2eac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:10.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.504+0000 7f3b2eac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:10.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.504+0000 7f3b2eac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:10.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.504+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b281daca0 con 0x7f3b2814be90 2024-03-25T17:35:10.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.525+0000 7f3b257fa640 1 -- 172.21.15.96:0/1343524870 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f3b20052500 con 0x7f3b2814be90 2024-03-25T17:35:10.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.525+0000 7f3b257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:10.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.525+0000 7f3b257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:10.526 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:10.527 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:10.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 10 librados: watch_flush enter 2024-03-25T17:35:10.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 10 librados: watch_flush exit 2024-03-25T17:35:10.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 10 client.4486.objecter successfully canceled tick 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 >> v1:172.21.15.203:6800/3611964061 conn(0x7f3b0c06de80 legacy=0x7f3b0c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 10 monclient: shutdown 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 >> v1:172.21.15.203:6789/0 conn(0x7f3b2814be90 legacy=0x7f3b28130de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 shutdown_connections 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 >> 172.21.15.96:0/1343524870 conn(0x7f3b280617a0 msgr2=0x7f3b280b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 shutdown_connections 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 -- 172.21.15.96:0/1343524870 wait complete. 2024-03-25T17:35:10.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:10.528+0000 7f3b2eac8640 1 librados: shutdown 2024-03-25T17:35:10.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:17.346 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:17.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 10 monclient: get_monmap_and_config 2024-03-25T17:35:17.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 10 monclient: build_initial_monmap 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 10 monclient: monmap: 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:17.458329+0000 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:17.458329+0000 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:17.458 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 1 Processor -- start 2024-03-25T17:35:17.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 1 -- start start 2024-03-25T17:35:17.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.457+0000 7f2ebc77b640 10 monclient: init 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb4142520 con 0x7f2eb41430c0 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40b28a0 con 0x7f2eb413f710 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40b2a30 con 0x7f2eb40a98f0 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:17.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.458+0000 7f2ebc77b640 10 monclient(hunting): authenticate will time out at 5675.740820s 2024-03-25T17:35:17.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.465+0000 7f2ebad18640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2eb41430c0 0x7f2eb414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47938/0 (socket says 172.21.15.96:47938) 2024-03-25T17:35:17.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.465+0000 7f2ebad18640 1 -- 172.21.15.96:0/327463988 learned_addr learned my addr 172.21.15.96:0/327463988 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820370819 0 0) 0x7f2eb4142520 con 0x7f2eb41430c0 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 10 monclient(hunting): my global_id is 4503 2024-03-25T17:35:17.477 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.476+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea0003520 con 0x7f2eb41430c0 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437681954 0 0) 0x7f2eb40b28a0 con 0x7f2eb413f710 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 10 monclient(hunting): my global_id is 4489 2024-03-25T17:35:17.490 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.489+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb4142520 con 0x7f2eb413f710 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595229152 0 0) 0x7f2eb40b2a30 con 0x7f2eb40a98f0 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 10 monclient(hunting): my global_id is 4490 2024-03-25T17:35:17.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.493+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb40b28a0 con 0x7f2eb40a98f0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819721739 0 0) 0x7f2ea0003520 con 0x7f2eb41430c0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 >> v1:172.21.15.96:6790/0 conn(0x7f2eb40a98f0 legacy=0x7f2eb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 >> v1:172.21.15.203:6789/0 conn(0x7f2eb413f710 legacy=0x7f2eb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: _finish_hunting 0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 monclient: found mon.noname-a 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb40b44c0 con 0x7f2eb41430c0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: _finish_auth 0 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: _check_auth_tickets 2024-03-25T17:35:17.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2eac001a10 con 0x7f2eb41430c0 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2ebc77b640 5 monclient: authenticate success, global_id 4503 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2ebc77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2eb9515640 10 monclient: dump: 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:17.505 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.503+0000 7f2ebc77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:17.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eac001b70 con 0x7f2eb41430c0 2024-03-25T17:35:17.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:17.509 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 1 -- 172.21.15.96:0/327463988 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2eac001f80 con 0x7f2eb41430c0 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2eb9515640 10 monclient: dump: 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2ebc77b640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:17.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.508+0000 7f2ebc77b640 4 set_mon_vals no callback set 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 10 monclient: shutdown 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 1 -- 172.21.15.96:0/327463988 >> v1:172.21.15.96:6789/0 conn(0x7f2eb41430c0 legacy=0x7f2eb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 1 -- 172.21.15.96:0/327463988 shutdown_connections 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 1 -- 172.21.15.96:0/327463988 >> 172.21.15.96:0/327463988 conn(0x7f2eb40b7ea0 msgr2=0x7f2eb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 1 -- 172.21.15.96:0/327463988 shutdown_connections 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.509+0000 7f2ebc77b640 1 -- 172.21.15.96:0/327463988 wait complete. 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.510+0000 7f2ebc77b640 10 monclient: build_initial_monmap 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.510+0000 7f2ebc77b640 10 monclient: monmap: 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:17.512 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 librados: starting msgr at 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 librados: starting objecter 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 Processor -- start 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 -- start start 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 librados: setting wanted keys 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 1 librados: calling monclient init 2024-03-25T17:35:17.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.511+0000 7f2ebc77b640 10 monclient: init 2024-03-25T17:35:17.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:17.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:17.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb4136e20 con 0x7f2eb40a98f0 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40b7940 con 0x7f2eb41430c0 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb41351a0 con 0x7f2eb413f710 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:17.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.512+0000 7f2ebc77b640 10 monclient(hunting): authenticate will time out at 5675.794456s 2024-03-25T17:35:17.518 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.517+0000 7f2ebad18640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2eb41430c0 0x7f2eb41e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:44460/0 (socket says 172.21.15.96:44460) 2024-03-25T17:35:17.518 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.517+0000 7f2ebad18640 1 -- 172.21.15.96:0/1296988376 learned_addr learned my addr 172.21.15.96:0/1296988376 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:17.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.519+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb413f710 2024-03-25T17:35:17.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.519+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:17.522 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.522+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb41430c0 2024-03-25T17:35:17.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.522+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:17.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.528+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb40a98f0 2024-03-25T17:35:17.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.528+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:17.534 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949717936 0 0) 0x7f2eb40b7940 con 0x7f2eb41430c0 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4492 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.533+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e98004eb0 con 0x7f2eb41430c0 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178814379 0 0) 0x7f2eb4136e20 con 0x7f2eb40a98f0 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4506 2024-03-25T17:35:17.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.534+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb40b7940 con 0x7f2eb40a98f0 2024-03-25T17:35:17.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398192933 0 0) 0x7f2eb41351a0 con 0x7f2eb413f710 2024-03-25T17:35:17.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:17.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:17.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:17.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T17:35:17.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.537+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb4136e20 con 0x7f2eb413f710 2024-03-25T17:35:17.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.539+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328806569 0 0) 0x7f2eb40b7940 con 0x7f2eb40a98f0 2024-03-25T17:35:17.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.539+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb41351a0 con 0x7f2eb40a98f0 2024-03-25T17:35:17.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.541+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2eb0001930 con 0x7f2eb40a98f0 2024-03-25T17:35:17.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.541+0000 7f2eaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:17.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.541+0000 7f2eaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:17.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.541+0000 7f2eaaffd640 10 monclient(hunting): dump: 2024-03-25T17:35:17.542 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.541+0000 7f2eaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:17.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2347569914 0 0) 0x7f2eb41351a0 con 0x7f2eb40a98f0 2024-03-25T17:35:17.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 >> v1:172.21.15.96:6790/0 conn(0x7f2eb413f710 legacy=0x7f2eb41db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:17.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 >> v1:172.21.15.203:6789/0 conn(0x7f2eb41430c0 legacy=0x7f2eb41e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:17.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:35:17.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 1 monclient: found mon.a 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb4135330 con 0x7f2eb40a98f0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 10 monclient: _finish_auth 0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2eaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 5 monclient: authenticate success, global_id 4506 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 10 monclient: _renew_subs 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb4135560 con 0x7f2eb40a98f0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 10 monclient: _renew_subs 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb413b490 con 0x7f2eb40a98f0 2024-03-25T17:35:17.553 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.551+0000 7f2ebc77b640 1 librados: init done 2024-03-25T17:35:17.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.552+0000 7f2ebc77b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:17.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.552+0000 7f2ebc77b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:17.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.552+0000 7f2ebc77b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:17.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.552+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb40b7ad0 con 0x7f2eb40a98f0 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb0003610 con 0x7f2eb40a98f0 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f2eb0001610 con 0x7f2eb40a98f0 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eaaffd640 10 monclient: dump: 2024-03-25T17:35:17.850 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:17.851 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.849+0000 7f2eb8d14640 4 set_mon_vals no callback set 2024-03-25T17:35:17.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.850+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f2eb0003b20 con 0x7f2eb40a98f0 2024-03-25T17:35:17.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.850+0000 7f2eb9d16640 10 monclient: get_auth_request con 0x7f2e9806ddf0 auth_method 0 2024-03-25T17:35:17.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.851+0000 7f2eaaffd640 10 client.4506.objecter ms_handle_connect 0x7f2e9806ddf0 2024-03-25T17:35:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f2eb0053610 con 0x7f2eb40a98f0 2024-03-25T17:35:17.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 10 client.4506.objecter ms_dispatch 0x7f2eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2eb004f120 con 0x7f2eb40a98f0 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:17.858 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:17.857+0000 7f2eaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:18.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.106+0000 7f2ebc77b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:18.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.106+0000 7f2ebc77b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:18.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.106+0000 7f2ebc77b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:18.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.106+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb40b7730 con 0x7f2eb40a98f0 2024-03-25T17:35:18.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.133+0000 7f2eaaffd640 1 -- 172.21.15.96:0/1296988376 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f2eb0051050 con 0x7f2eb40a98f0 2024-03-25T17:35:18.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.133+0000 7f2eaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:18.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.133+0000 7f2eaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:18.135 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:18.135 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:18.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 10 librados: watch_flush enter 2024-03-25T17:35:18.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 10 librados: watch_flush exit 2024-03-25T17:35:18.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T17:35:18.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 10 client.4506.objecter successfully canceled tick 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 >> v1:172.21.15.203:6800/3611964061 conn(0x7f2e9806ddf0 legacy=0x7f2e980702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 10 monclient: shutdown 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 >> v1:172.21.15.96:6789/0 conn(0x7f2eb40a98f0 legacy=0x7f2eb41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 shutdown_connections 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 >> 172.21.15.96:0/1296988376 conn(0x7f2eb40b7ea0 msgr2=0x7f2eb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.136+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 shutdown_connections 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.137+0000 7f2ebc77b640 1 -- 172.21.15.96:0/1296988376 wait complete. 2024-03-25T17:35:18.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:18.137+0000 7f2ebc77b640 1 librados: shutdown 2024-03-25T17:35:18.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:21.022 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:45.249 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.034 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:45.260 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.060 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:06:45.286 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.084 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.299 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.090 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:45.316 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.092 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:45.319 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.094 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.309 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.115 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:06:45.341 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:21.117 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.332 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:24.955 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:25.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.073+0000 7f30522e4640 10 monclient: get_monmap_and_config 2024-03-25T17:35:25.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.073+0000 7f30522e4640 10 monclient: build_initial_monmap 2024-03-25T17:35:25.074 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.073+0000 7f30522e4640 10 monclient: monmap: 2024-03-25T17:35:25.074 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:25.074 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:25.075052+0000 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:25.075052+0000 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:25.075 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.074+0000 7f30522e4640 1 Processor -- start 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.074+0000 7f30522e4640 1 -- start start 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.074+0000 7f30522e4640 10 monclient: init 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-c con 0x7f304c0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-a con 0x7f304c0b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:25.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-b con 0x7f304c14be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:25.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c0af940 con 0x7f304c0b1730 2024-03-25T17:35:25.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c0b0a70 con 0x7f304c0b4d60 2024-03-25T17:35:25.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c0b0d50 con 0x7f304c14be90 2024-03-25T17:35:25.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:25.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.075+0000 7f30522e4640 10 monclient(hunting): authenticate will time out at 5683.357451s 2024-03-25T17:35:25.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.090+0000 7f3050881640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f304c14be90 0x7f304c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:57926/0 (socket says 172.21.15.96:57926) 2024-03-25T17:35:25.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.090+0000 7f3050881640 1 -- 172.21.15.96:0/1209259488 learned_addr learned my addr 172.21.15.96:0/1209259488 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:25.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681507736 0 0) 0x7f304c0b0d50 con 0x7f304c14be90 2024-03-25T17:35:25.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 10 monclient(hunting): my global_id is 4496 2024-03-25T17:35:25.101 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.099+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034003520 con 0x7f304c14be90 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636427421 0 0) 0x7f304c0af940 con 0x7f304c0b1730 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 10 monclient(hunting): my global_id is 4509 2024-03-25T17:35:25.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.112+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c0b0d50 con 0x7f304c0b1730 2024-03-25T17:35:25.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374669066 0 0) 0x7f304c0b0a70 con 0x7f304c0b4d60 2024-03-25T17:35:25.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.115 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 10 monclient(hunting): my global_id is 4495 2024-03-25T17:35:25.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.114+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c0af940 con 0x7f304c0b4d60 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3361983330 0 0) 0x7f3034003520 con 0x7f304c14be90 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 >> v1:172.21.15.203:6789/0 conn(0x7f304c0b4d60 legacy=0x7f304c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 >> v1:172.21.15.96:6789/0 conn(0x7f304c0b1730 legacy=0x7f304c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: _finish_hunting 0 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 monclient: found mon.noname-b 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0b1030 con 0x7f304c14be90 2024-03-25T17:35:25.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: _finish_auth 0 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: _check_auth_tickets 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3044001a10 con 0x7f304c14be90 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f304affd640 10 monclient: dump: 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.130 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.131 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f30522e4640 5 monclient: authenticate success, global_id 4496 2024-03-25T17:35:25.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.128+0000 7f30522e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:25.150 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3044001b70 con 0x7f304c14be90 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 1 -- 172.21.15.96:0/1209259488 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3044001f80 con 0x7f304c14be90 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f30522e4640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f30522e4640 4 set_mon_vals no callback set 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.150+0000 7f304affd640 10 monclient: dump: 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.151 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 10 monclient: shutdown 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 1 -- 172.21.15.96:0/1209259488 >> v1:172.21.15.96:6790/0 conn(0x7f304c14be90 legacy=0x7f304c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 1 -- 172.21.15.96:0/1209259488 shutdown_connections 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 1 -- 172.21.15.96:0/1209259488 >> 172.21.15.96:0/1209259488 conn(0x7f304c0617a0 msgr2=0x7f304c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 1 -- 172.21.15.96:0/1209259488 shutdown_connections 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.151+0000 7f30522e4640 1 -- 172.21.15.96:0/1209259488 wait complete. 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 10 monclient: build_initial_monmap 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 10 monclient: monmap: 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:25.153 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 librados: starting msgr at 2024-03-25T17:35:25.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 librados: starting objecter 2024-03-25T17:35:25.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 Processor -- start 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 -- start start 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 librados: setting wanted keys 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 1 librados: calling monclient init 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.152+0000 7f30522e4640 10 monclient: init 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-b con 0x7f304c0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-a con 0x7f304c0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient(hunting): picked mon.noname-c con 0x7f304c14be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c140760 con 0x7f304c0b4d60 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c140380 con 0x7f304c0b1730 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c140510 con 0x7f304c14be90 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:25.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.153+0000 7f30522e4640 10 monclient(hunting): authenticate will time out at 5683.435696s 2024-03-25T17:35:25.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.162+0000 7f304bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f304c0b1730 0x7f304c1dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:55178/0 (socket says 172.21.15.96:55178) 2024-03-25T17:35:25.163 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.162+0000 7f304bfff640 1 -- 172.21.15.96:0/2302914249 learned_addr learned my addr 172.21.15.96:0/2302914249 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:25.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.166+0000 7f3048ff9640 10 client.?.objecter ms_handle_connect 0x7f304c0b1730 2024-03-25T17:35:25.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.167+0000 7f3048ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:25.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.173+0000 7f3048ff9640 10 client.?.objecter ms_handle_connect 0x7f304c14be90 2024-03-25T17:35:25.174 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.173+0000 7f3048ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:25.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.175+0000 7f3048ff9640 10 client.?.objecter ms_handle_connect 0x7f304c0b4d60 2024-03-25T17:35:25.176 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.175+0000 7f3048ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959364473 0 0) 0x7f304c140380 con 0x7f304c0b1730 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T17:35:25.177 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.176+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3038004eb0 con 0x7f304c0b1730 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385059818 0 0) 0x7f304c140760 con 0x7f304c0b4d60 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 10 monclient(hunting): my global_id is 4512 2024-03-25T17:35:25.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.185+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c140380 con 0x7f304c0b4d60 2024-03-25T17:35:25.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837727564 0 0) 0x7f304c140510 con 0x7f304c14be90 2024-03-25T17:35:25.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:25.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:25.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:25.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 10 monclient(hunting): my global_id is 4499 2024-03-25T17:35:25.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.187+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c140760 con 0x7f304c14be90 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1277500124 0 0) 0x7f3038004eb0 con 0x7f304c0b1730 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304c140510 con 0x7f304c0b1730 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f303c001a10 con 0x7f304c0b1730 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 10 monclient(hunting): dump: 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.199 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.200 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.198+0000 7f3048ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:25.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1797696807 0 0) 0x7f304c140380 con 0x7f304c0b4d60 2024-03-25T17:35:25.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3038004eb0 con 0x7f304c0b4d60 2024-03-25T17:35:25.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3040001970 con 0x7f304c0b4d60 2024-03-25T17:35:25.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.201+0000 7f3048ff9640 10 monclient(hunting): dump: 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.203 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903838866 0 0) 0x7f304c140760 con 0x7f304c14be90 2024-03-25T17:35:25.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304c140380 con 0x7f304c14be90 2024-03-25T17:35:25.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3044001e80 con 0x7f304c14be90 2024-03-25T17:35:25.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.209+0000 7f3048ff9640 10 monclient(hunting): dump: 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.211 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 741863978 0 0) 0x7f3038004eb0 con 0x7f304c0b4d60 2024-03-25T17:35:25.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 >> v1:172.21.15.96:6790/0 conn(0x7f304c14be90 legacy=0x7f304c1e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 >> v1:172.21.15.203:6789/0 conn(0x7f304c0b1730 legacy=0x7f304c1dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 1 monclient: found mon.a 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c137530 con 0x7f304c0b4d60 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 10 monclient: _finish_auth 0 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f3048ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 5 monclient: authenticate success, global_id 4512 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 10 monclient: _renew_subs 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c138070 con 0x7f304c0b4d60 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 10 monclient: _renew_subs 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:25.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c145000 con 0x7f304c0b4d60 2024-03-25T17:35:25.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.221+0000 7f30522e4640 1 librados: init done 2024-03-25T17:35:25.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.222+0000 7f30522e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:25.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.222+0000 7f30522e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:25.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.222+0000 7f30522e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:25.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.222+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c1408f0 con 0x7f304c0b4d60 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30400035f0 con 0x7f304c0b4d60 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3040003900 con 0x7f304c0b4d60 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f3048ff9640 10 monclient: dump: 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:25.529 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:25.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.528+0000 7f304a7fc640 4 set_mon_vals no callback set 2024-03-25T17:35:25.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.529+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f30400198d0 con 0x7f304c0b4d60 2024-03-25T17:35:25.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.529+0000 7f304bfff640 10 monclient: get_auth_request con 0x7f303806e110 auth_method 0 2024-03-25T17:35:25.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.530+0000 7f3048ff9640 10 client.4512.objecter ms_handle_connect 0x7f303806e110 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f3040053530 con 0x7f304c0b4d60 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 10 client.4512.objecter ms_dispatch 0x7f304c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f304004ee70 con 0x7f304c0b4d60 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:25.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.535+0000 7f3048ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:25.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.791+0000 7f30522e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:25.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.791+0000 7f30522e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:25.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.791+0000 7f30522e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:25.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.791+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f304c0627b0 con 0x7f304c0b4d60 2024-03-25T17:35:25.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.813+0000 7f3048ff9640 1 -- 172.21.15.96:0/2302914249 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f3040050f70 con 0x7f304c0b4d60 2024-03-25T17:35:25.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.813+0000 7f3048ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:25.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.813+0000 7f3048ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:25.814 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:25.814 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 10 librados: watch_flush enter 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 10 librados: watch_flush exit 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 10 client.4512.objecter successfully canceled tick 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 >> v1:172.21.15.203:6800/3611964061 conn(0x7f303806e110 legacy=0x7f30380705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 10 monclient: shutdown 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 >> v1:172.21.15.96:6789/0 conn(0x7f304c0b4d60 legacy=0x7f304c1db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 shutdown_connections 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.815+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 >> 172.21.15.96:0/2302914249 conn(0x7f304c0617a0 msgr2=0x7f304c13c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.816+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 shutdown_connections 2024-03-25T17:35:25.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.816+0000 7f30522e4640 1 -- 172.21.15.96:0/2302914249 wait complete. 2024-03-25T17:35:25.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:25.816+0000 7f30522e4640 1 librados: shutdown 2024-03-25T17:35:25.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:32.635 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:32.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.747+0000 7f107cbc9640 10 monclient: get_monmap_and_config 2024-03-25T17:35:32.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.747+0000 7f107cbc9640 10 monclient: build_initial_monmap 2024-03-25T17:35:32.748 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.747+0000 7f107cbc9640 10 monclient: monmap: 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:32.749224+0000 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:32.749224+0000 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:32.749 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:32.750 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.748+0000 7f107cbc9640 1 Processor -- start 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.748+0000 7f107cbc9640 1 -- start start 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.748+0000 7f107cbc9640 10 monclient: init 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-c con 0x7f10780a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-a con 0x7f107813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-b con 0x7f10781430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078142520 con 0x7f107813f710 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780b28a0 con 0x7f10780a98f0 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780b2a30 con 0x7f10781430c0 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:32.751 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.749+0000 7f107cbc9640 10 monclient(hunting): authenticate will time out at 5691.031394s 2024-03-25T17:35:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.756+0000 7f1076d9d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f10781430c0 0x7f107814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:34650/0 (socket says 172.21.15.96:34650) 2024-03-25T17:35:32.756 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.756+0000 7f1076d9d640 1 -- 172.21.15.96:0/729683738 learned_addr learned my addr 172.21.15.96:0/729683738 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005618384 0 0) 0x7f10780b2a30 con 0x7f10781430c0 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 10 monclient(hunting): _negotiate 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 10 monclient(hunting): my global_id is 4502 2024-03-25T17:35:32.765 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.764+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064003520 con 0x7f10781430c0 2024-03-25T17:35:32.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273293322 0 0) 0x7f10780b28a0 con 0x7f10780a98f0 2024-03-25T17:35:32.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 10 monclient(hunting): _negotiate 2024-03-25T17:35:32.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:32.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:32.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 10 monclient(hunting): my global_id is 4501 2024-03-25T17:35:32.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.773+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10780b2a30 con 0x7f10780a98f0 2024-03-25T17:35:32.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183254584 0 0) 0x7f1078142520 con 0x7f107813f710 2024-03-25T17:35:32.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 10 monclient(hunting): _negotiate 2024-03-25T17:35:32.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:32.779 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:32.779 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 10 monclient(hunting): my global_id is 4515 2024-03-25T17:35:32.779 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.778+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10780b28a0 con 0x7f107813f710 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645437350 0 0) 0x7f1064003520 con 0x7f10781430c0 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 >> v1:172.21.15.203:6789/0 conn(0x7f10780a98f0 legacy=0x7f10780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 >> v1:172.21.15.96:6789/0 conn(0x7f107813f710 legacy=0x7f10781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: _finish_hunting 0 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 monclient: found mon.noname-b 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10780b44c0 con 0x7f10781430c0 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: _finish_auth 0 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: _check_auth_tickets 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:32.782 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1068001690 con 0x7f10781430c0 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107cbc9640 5 monclient: authenticate success, global_id 4502 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107cbc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107559a640 10 monclient: dump: 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:32.783 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:32.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.781+0000 7f107cbc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10680017f0 con 0x7f10781430c0 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 1 -- 172.21.15.96:0/729683738 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1068001c00 con 0x7f10781430c0 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107559a640 10 monclient: dump: 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:32.800 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107cbc9640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:32.801 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.799+0000 7f107cbc9640 4 set_mon_vals no callback set 2024-03-25T17:35:32.802 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 10 monclient: shutdown 2024-03-25T17:35:32.802 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 1 -- 172.21.15.96:0/729683738 >> v1:172.21.15.96:6790/0 conn(0x7f10781430c0 legacy=0x7f107814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 1 -- 172.21.15.96:0/729683738 shutdown_connections 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 1 -- 172.21.15.96:0/729683738 >> 172.21.15.96:0/729683738 conn(0x7f10780b7ea0 msgr2=0x7f10780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 1 -- 172.21.15.96:0/729683738 shutdown_connections 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.800+0000 7f107cbc9640 1 -- 172.21.15.96:0/729683738 wait complete. 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.801+0000 7f107cbc9640 10 monclient: build_initial_monmap 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.801+0000 7f107cbc9640 10 monclient: monmap: 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:32.803 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.801+0000 7f107cbc9640 1 librados: starting msgr at 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.801+0000 7f107cbc9640 1 librados: starting objecter 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 1 Processor -- start 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 1 -- start start 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 1 librados: setting wanted keys 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 1 librados: calling monclient init 2024-03-25T17:35:32.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient: init 2024-03-25T17:35:32.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:32.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:32.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-c con 0x7f10780a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:32.805 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-b con 0x7f107813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.802+0000 7f107cbc9640 10 monclient(hunting): picked mon.noname-a con 0x7f10781430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.803+0000 7f107cbc9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107813c9b0 con 0x7f10781430c0 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.803+0000 7f107cbc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10781397c0 con 0x7f107813f710 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.803+0000 7f107cbc9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078139950 con 0x7f10780a98f0 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.803+0000 7f107cbc9640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:32.806 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.803+0000 7f107cbc9640 10 monclient(hunting): authenticate will time out at 5691.085013s 2024-03-25T17:35:32.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.814+0000 7f1075d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f107813f710 0x7f10781dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:60090/0 (socket says 172.21.15.96:60090) 2024-03-25T17:35:32.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.814+0000 7f1075d9b640 1 -- 172.21.15.96:0/1461455768 learned_addr learned my addr 172.21.15.96:0/1461455768 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:32.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.815+0000 7f106effd640 10 client.?.objecter ms_handle_connect 0x7f107813f710 2024-03-25T17:35:32.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.815+0000 7f106effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:32.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.816+0000 7f106effd640 10 client.?.objecter ms_handle_connect 0x7f10781430c0 2024-03-25T17:35:32.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.816+0000 7f106effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:32.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680587049 0 0) 0x7f107813c9b0 con 0x7f10781430c0 2024-03-25T17:35:32.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:32.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:32.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:32.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 10 monclient(hunting): my global_id is 4518 2024-03-25T17:35:32.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.823+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058004eb0 con 0x7f10781430c0 2024-03-25T17:35:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.827+0000 7f106effd640 10 client.?.objecter ms_handle_connect 0x7f10780a98f0 2024-03-25T17:35:32.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.827+0000 7f106effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059306258 0 0) 0x7f1058004eb0 con 0x7f10781430c0 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f107813c9b0 con 0x7f10781430c0 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1068001b00 con 0x7f10781430c0 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 10 monclient(hunting): dump: 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:32.852 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:32.853 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:32.853 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:32.853 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.851+0000 7f106effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.909+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1413672373 0 0) 0x7f107813c9b0 con 0x7f10781430c0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 >> v1:172.21.15.96:6790/0 conn(0x7f10780a98f0 legacy=0x7f1078130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 >> v1:172.21.15.203:6789/0 conn(0x7f107813f710 legacy=0x7f10781dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 10 monclient: _finish_hunting 0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 1 monclient: found mon.a 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078131680 con 0x7f10781430c0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 10 monclient: _finish_auth 0 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 10 monclient: _check_auth_tickets 2024-03-25T17:35:32.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f106effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 5 monclient: authenticate success, global_id 4518 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 10 monclient: _renew_subs 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1078131950 con 0x7f10781430c0 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 10 monclient: _renew_subs 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078135a30 con 0x7f10781430c0 2024-03-25T17:35:32.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.910+0000 7f107cbc9640 1 librados: init done 2024-03-25T17:35:32.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.911+0000 7f107cbc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:32.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.911+0000 7f107cbc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:32.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.911+0000 7f107cbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:32.913 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:32.911+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107813c810 con 0x7f10781430c0 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1068001ce0 con 0x7f10781430c0 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1068004a90 con 0x7f10781430c0 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 10 monclient: dump: 2024-03-25T17:35:33.199 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f1074d99640 4 set_mon_vals no callback set 2024-03-25T17:35:33.200 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.198+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f106801aa60 con 0x7f10781430c0 2024-03-25T17:35:33.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.199+0000 7f107659c640 10 monclient: get_auth_request con 0x7f105806dbd0 auth_method 0 2024-03-25T17:35:33.201 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.199+0000 7f106effd640 10 client.4518.objecter ms_handle_connect 0x7f105806dbd0 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.208+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f10680545f0 con 0x7f10781430c0 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 10 client.4518.objecter ms_dispatch 0x7f1078000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106804ff30 con 0x7f10781430c0 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:33.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.209+0000 7f106effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:33.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.454+0000 7f107cbc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:33.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.454+0000 7f107cbc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:33.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.454+0000 7f107cbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:33.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.454+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f107813cbf0 con 0x7f10781430c0 2024-03-25T17:35:33.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.482+0000 7f106effd640 1 -- 172.21.15.96:0/1461455768 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f107813cbf0 con 0x7f10781430c0 2024-03-25T17:35:33.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.482+0000 7f106effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:33.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.482+0000 7f106effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:33.483 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:33.483 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 10 librados: watch_flush enter 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 10 librados: watch_flush exit 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 10 client.4518.objecter successfully canceled tick 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 >> v1:172.21.15.203:6800/3611964061 conn(0x7f105806dbd0 legacy=0x7f1058070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 10 monclient: shutdown 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 >> v1:172.21.15.96:6789/0 conn(0x7f10781430c0 legacy=0x7f10781e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 shutdown_connections 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 >> 172.21.15.96:0/1461455768 conn(0x7f10780b7ea0 msgr2=0x7f107814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:33.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 shutdown_connections 2024-03-25T17:35:33.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 -- 172.21.15.96:0/1461455768 wait complete. 2024-03-25T17:35:33.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:33.484+0000 7f107cbc9640 1 librados: shutdown 2024-03-25T17:35:33.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:40.305 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.418+0000 7f1142a74640 10 monclient: get_monmap_and_config 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.418+0000 7f1142a74640 10 monclient: build_initial_monmap 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.418+0000 7f1142a74640 10 monclient: monmap: 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:40.419022+0000 2024-03-25T17:35:40.418 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:40.419022+0000 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:40.419 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.419+0000 7f1142a74640 1 Processor -- start 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.419+0000 7f1142a74640 1 -- start start 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.419+0000 7f1142a74640 10 monclient: init 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-a con 0x7f113c0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-b con 0x7f113c13f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:40.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-c con 0x7f113c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:40.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c142520 con 0x7f113c0a98f0 2024-03-25T17:35:40.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0b28a0 con 0x7f113c1430c0 2024-03-25T17:35:40.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0b2a30 con 0x7f113c13f710 2024-03-25T17:35:40.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:40.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.420+0000 7f1142a74640 10 monclient(hunting): authenticate will time out at 5698.701244s 2024-03-25T17:35:40.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.425+0000 7f113bfff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f113c13f710 0x7f113c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:45246/0 (socket says 172.21.15.96:45246) 2024-03-25T17:35:40.425 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.425+0000 7f113bfff640 1 -- 172.21.15.96:0/1715544089 learned_addr learned my addr 172.21.15.96:0/1715544089 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84945200 0 0) 0x7f113c0b28a0 con 0x7f113c1430c0 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T17:35:40.439 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.439+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112c003520 con 0x7f113c1430c0 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085876589 0 0) 0x7f113c0b2a30 con 0x7f113c13f710 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T17:35:40.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.441+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c0b28a0 con 0x7f113c13f710 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346367421 0 0) 0x7f113c142520 con 0x7f113c0a98f0 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4521 2024-03-25T17:35:40.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.443+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c0b2a30 con 0x7f113c0a98f0 2024-03-25T17:35:40.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294476621 0 0) 0x7f112c003520 con 0x7f113c1430c0 2024-03-25T17:35:40.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 >> v1:172.21.15.96:6790/0 conn(0x7f113c13f710 legacy=0x7f113c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:40.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 >> v1:172.21.15.96:6789/0 conn(0x7f113c0a98f0 legacy=0x7f113c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:40.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:40.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 monclient: found mon.noname-c 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0b44c0 con 0x7f113c1430c0 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1134001a10 con 0x7f113c1430c0 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f113b7fe640 10 monclient: dump: 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:40.450 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f1142a74640 5 monclient: authenticate success, global_id 4507 2024-03-25T17:35:40.451 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.449+0000 7f1142a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134001b70 con 0x7f113c1430c0 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 1 -- 172.21.15.96:0/1715544089 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1134001f80 con 0x7f113c1430c0 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f113b7fe640 10 monclient: dump: 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:40.456 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:40.457 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:40.457 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f1142a74640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:40.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.456+0000 7f1142a74640 4 set_mon_vals no callback set 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 10 monclient: shutdown 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 1 -- 172.21.15.96:0/1715544089 >> v1:172.21.15.203:6789/0 conn(0x7f113c1430c0 legacy=0x7f113c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 1 -- 172.21.15.96:0/1715544089 shutdown_connections 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 1 -- 172.21.15.96:0/1715544089 >> 172.21.15.96:0/1715544089 conn(0x7f113c0b7ea0 msgr2=0x7f113c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 1 -- 172.21.15.96:0/1715544089 shutdown_connections 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.457+0000 7f1142a74640 1 -- 172.21.15.96:0/1715544089 wait complete. 2024-03-25T17:35:40.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 10 monclient: build_initial_monmap 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 10 monclient: monmap: 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 librados: starting msgr at 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 librados: starting objecter 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 Processor -- start 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 -- start start 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 librados: setting wanted keys 2024-03-25T17:35:40.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 1 librados: calling monclient init 2024-03-25T17:35:40.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.458+0000 7f1142a74640 10 monclient: init 2024-03-25T17:35:40.460 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-c con 0x7f113c0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-a con 0x7f113c13f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient(hunting): picked mon.noname-b con 0x7f113c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1daca0 con 0x7f113c13f710 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c138ab0 con 0x7f113c1430c0 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0aba90 con 0x7f113c0a98f0 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:40.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.459+0000 7f1142a74640 10 monclient(hunting): authenticate will time out at 5698.740800s 2024-03-25T17:35:40.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.474+0000 7f1140810640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f113c0a98f0 0x7f113c1383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:45264/0 (socket says 172.21.15.96:45264) 2024-03-25T17:35:40.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.474+0000 7f1140810640 1 -- 172.21.15.96:0/1663333312 learned_addr learned my addr 172.21.15.96:0/1663333312 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:40.476 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.476+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c0a98f0 2024-03-25T17:35:40.476 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.476+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:40.476 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.477+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c1430c0 2024-03-25T17:35:40.476 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.477+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:40.483 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.483+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c13f710 2024-03-25T17:35:40.483 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.483+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:40.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349453646 0 0) 0x7f113c138ab0 con 0x7f113c1430c0 2024-03-25T17:35:40.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.485 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.485 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T17:35:40.485 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.485+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c1430c0 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 650071284 0 0) 0x7f1120004eb0 con 0x7f113c1430c0 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c138ab0 con 0x7f113c1430c0 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1134001e80 con 0x7f113c1430c0 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 10 monclient(hunting): dump: 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:40.495 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:40.496 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:40.496 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:40.496 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:40.496 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.495+0000 7f11397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:40.498 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809108135 0 0) 0x7f113c1daca0 con 0x7f113c13f710 2024-03-25T17:35:40.498 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 10 monclient(hunting): my global_id is 4524 2024-03-25T17:35:40.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.499+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c13f710 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207612958 0 0) 0x7f113c0aba90 con 0x7f113c0a98f0 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 10 monclient(hunting): my global_id is 4511 2024-03-25T17:35:40.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.503+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c1daca0 con 0x7f113c0a98f0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3458003619 0 0) 0x7f113c138ab0 con 0x7f113c1430c0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 >> v1:172.21.15.96:6790/0 conn(0x7f113c0a98f0 legacy=0x7f113c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 >> v1:172.21.15.96:6789/0 conn(0x7f113c13f710 legacy=0x7f113c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 1 monclient: found mon.b 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c1d41e0 con 0x7f113c1430c0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:40.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f11397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 5 monclient: authenticate success, global_id 4510 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 10 monclient: _renew_subs 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c1db840 con 0x7f113c1430c0 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 10 monclient: _renew_subs 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c138f00 con 0x7f113c1430c0 2024-03-25T17:35:40.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.506+0000 7f1142a74640 1 librados: init done 2024-03-25T17:35:40.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.507+0000 7f1142a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:40.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.507+0000 7f1142a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:40.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.507+0000 7f1142a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:40.508 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.507+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c1dae50 con 0x7f113c1430c0 2024-03-25T17:35:40.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.515+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134002060 con 0x7f113c1430c0 2024-03-25T17:35:40.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.515+0000 7f11397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:40.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.515+0000 7f113affd640 4 set_mon_vals no callback set 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.793+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1134003c20 con 0x7f113c1430c0 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.793+0000 7f11397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.793+0000 7f11397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.793+0000 7f11397fa640 10 monclient: dump: 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.793 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:40.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.795+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1134019bf0 con 0x7f113c1430c0 2024-03-25T17:35:40.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.795+0000 7f1140810640 10 monclient: get_auth_request con 0x7f112006de80 auth_method 0 2024-03-25T17:35:40.795 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.796+0000 7f11397fa640 10 client.4510.objecter ms_handle_connect 0x7f112006de80 2024-03-25T17:35:40.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.797+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f11340537b0 con 0x7f113c1430c0 2024-03-25T17:35:40.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.797+0000 7f11397fa640 10 client.4510.objecter ms_dispatch 0x7f113c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:40.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.797+0000 7f11397fa640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:40.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.797+0000 7f11397fa640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:40.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.797+0000 7f11397fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T17:35:40.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.800+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f113404f0f0 con 0x7f113c1430c0 2024-03-25T17:35:40.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.800+0000 7f11397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:40.800 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:40.800+0000 7f11397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:41.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.027+0000 7f1142a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:41.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.027+0000 7f1142a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:41.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.027+0000 7f1142a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:35:41.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.027+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113c1dba80 con 0x7f113c1430c0 2024-03-25T17:35:41.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.122+0000 7f11397fa640 1 -- 172.21.15.96:0/1663333312 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f11340511f0 con 0x7f113c1430c0 2024-03-25T17:35:41.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.122+0000 7f11397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:41.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.122+0000 7f11397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:41.122 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:41.122 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.124+0000 7f1142a74640 10 librados: watch_flush enter 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 10 librados: watch_flush exit 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 10 client.4510.objecter successfully canceled tick 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 >> v1:172.21.15.203:6800/3611964061 conn(0x7f112006de80 legacy=0x7f1120070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 10 monclient: shutdown 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 >> v1:172.21.15.203:6789/0 conn(0x7f113c1430c0 legacy=0x7f113c1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 shutdown_connections 2024-03-25T17:35:41.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 >> 172.21.15.96:0/1663333312 conn(0x7f113c0b7ea0 msgr2=0x7f113c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:41.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 shutdown_connections 2024-03-25T17:35:41.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 -- 172.21.15.96:0/1663333312 wait complete. 2024-03-25T17:35:41.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:41.125+0000 7f1142a74640 1 librados: shutdown 2024-03-25T17:35:41.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:46.071 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:10.297 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.075 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:10.302 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.098 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:10.325 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.131 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:10.346 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.132 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:10.359 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.134 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:10.350 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.144 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:10.371 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.169 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:10.384 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:46.180 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:10.406 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:47.943 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:48.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.056+0000 7f32269d6640 10 monclient: get_monmap_and_config 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.056+0000 7f32269d6640 10 monclient: build_initial_monmap 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.056+0000 7f32269d6640 10 monclient: monmap: 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:48.058131+0000 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:48.058131+0000 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:48.058 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.057+0000 7f32269d6640 1 Processor -- start 2024-03-25T17:35:48.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.057+0000 7f32269d6640 1 -- start start 2024-03-25T17:35:48.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.057+0000 7f32269d6640 10 monclient: init 2024-03-25T17:35:48.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:48.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-a con 0x7f32200a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-b con 0x7f322013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-c con 0x7f32201430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.058+0000 7f32269d6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3220142520 con 0x7f32200a98f0 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.059+0000 7f32269d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200b28a0 con 0x7f32201430c0 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.059+0000 7f32269d6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200b2a30 con 0x7f322013f710 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.059+0000 7f32269d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:48.061 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.059+0000 7f32269d6640 10 monclient(hunting): authenticate will time out at 5706.341061s 2024-03-25T17:35:48.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.070+0000 7f321f7fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f322013f710 0x7f32201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:40022/0 (socket says 172.21.15.96:40022) 2024-03-25T17:35:48.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.070+0000 7f321f7fe640 1 -- 172.21.15.96:0/3075435072 learned_addr learned my addr 172.21.15.96:0/3075435072 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379392776 0 0) 0x7f32200b2a30 con 0x7f322013f710 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 10 monclient(hunting): my global_id is 4514 2024-03-25T17:35:48.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.076+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208003520 con 0x7f322013f710 2024-03-25T17:35:48.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377852465 0 0) 0x7f32200b28a0 con 0x7f32201430c0 2024-03-25T17:35:48.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:48.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:48.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:48.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 10 monclient(hunting): my global_id is 4513 2024-03-25T17:35:48.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.081+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32200b2a30 con 0x7f32201430c0 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466999309 0 0) 0x7f32200b2a30 con 0x7f32201430c0 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 >> v1:172.21.15.96:6790/0 conn(0x7f322013f710 legacy=0x7f32201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 >> v1:172.21.15.96:6789/0 conn(0x7f32200a98f0 legacy=0x7f32200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: _finish_hunting 0 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 monclient: found mon.noname-c 2024-03-25T17:35:48.089 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32200b44c0 con 0x7f32201430c0 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: _finish_auth 0 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: _check_auth_tickets 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f32180019f0 con 0x7f32201430c0 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f32269d6640 5 monclient: authenticate success, global_id 4513 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f32269d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f321effd640 10 monclient: dump: 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.090 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.088+0000 7f32269d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3218001b50 con 0x7f32201430c0 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 1 -- 172.21.15.96:0/3075435072 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3218001f20 con 0x7f32201430c0 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f32269d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f32269d6640 4 set_mon_vals no callback set 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:48.103 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.102+0000 7f321effd640 10 monclient: dump: 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:48.104 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.105 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 10 monclient: shutdown 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 1 -- 172.21.15.96:0/3075435072 >> v1:172.21.15.203:6789/0 conn(0x7f32201430c0 legacy=0x7f322014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 1 -- 172.21.15.96:0/3075435072 shutdown_connections 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 1 -- 172.21.15.96:0/3075435072 >> 172.21.15.96:0/3075435072 conn(0x7f32200b7ea0 msgr2=0x7f32200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 1 -- 172.21.15.96:0/3075435072 shutdown_connections 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.103+0000 7f32269d6640 1 -- 172.21.15.96:0/3075435072 wait complete. 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.104+0000 7f32269d6640 10 monclient: build_initial_monmap 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.104+0000 7f32269d6640 10 monclient: monmap: 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:48.106 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.104+0000 7f32269d6640 1 librados: starting msgr at 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.104+0000 7f32269d6640 1 librados: starting objecter 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.105+0000 7f32269d6640 1 Processor -- start 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.105+0000 7f32269d6640 1 -- start start 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.105+0000 7f32269d6640 1 librados: setting wanted keys 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.105+0000 7f32269d6640 1 librados: calling monclient init 2024-03-25T17:35:48.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.105+0000 7f32269d6640 10 monclient: init 2024-03-25T17:35:48.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:48.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:35:48.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-b con 0x7f32200a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:48.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-c con 0x7f322013f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:48.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient(hunting): picked mon.noname-a con 0x7f32201430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:48.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3220137e20 con 0x7f32201430c0 2024-03-25T17:35:48.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200b1720 con 0x7f32200a98f0 2024-03-25T17:35:48.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3220138710 con 0x7f322013f710 2024-03-25T17:35:48.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:48.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.106+0000 7f32269d6640 10 monclient(hunting): authenticate will time out at 5706.388226s 2024-03-25T17:35:48.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.115+0000 7f321ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f32200a98f0 0x7f3220137710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:42138/0 (socket says 172.21.15.96:42138) 2024-03-25T17:35:48.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.115+0000 7f321ffff640 1 -- 172.21.15.96:0/2038265347 learned_addr learned my addr 172.21.15.96:0/2038265347 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:48.117 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.117+0000 7f321cff9640 10 client.?.objecter ms_handle_connect 0x7f32201430c0 2024-03-25T17:35:48.118 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.117+0000 7f321cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:48.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.119+0000 7f321cff9640 10 client.?.objecter ms_handle_connect 0x7f32200a98f0 2024-03-25T17:35:48.119 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.119+0000 7f321cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:48.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.122+0000 7f321cff9640 10 client.?.objecter ms_handle_connect 0x7f322013f710 2024-03-25T17:35:48.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.122+0000 7f321cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:48.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831213760 0 0) 0x7f32200b1720 con 0x7f32200a98f0 2024-03-25T17:35:48.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:48.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:48.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:48.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T17:35:48.126 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.125+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320c004eb0 con 0x7f32200a98f0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338854651 0 0) 0x7f3220137e20 con 0x7f32201430c0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): my global_id is 4530 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32200b1720 con 0x7f32201430c0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419799411 0 0) 0x7f320c004eb0 con 0x7f32200a98f0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3220137e20 con 0x7f32200a98f0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3210001a30 con 0x7f32200a98f0 2024-03-25T17:35:48.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): dump: 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.136+0000 7f321cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:48.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110140531 0 0) 0x7f3220138710 con 0x7f322013f710 2024-03-25T17:35:48.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:48.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:48.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:48.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T17:35:48.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.138+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320c004eb0 con 0x7f322013f710 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102732063 0 0) 0x7f32200b1720 con 0x7f32201430c0 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3220138710 con 0x7f32201430c0 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3218001e60 con 0x7f32201430c0 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.143+0000 7f321cff9640 10 monclient(hunting): dump: 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.144 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:48.145 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:48.145 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:48.145 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.151 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1423283481 0 0) 0x7f3220138710 con 0x7f32201430c0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 >> v1:172.21.15.96:6790/0 conn(0x7f322013f710 legacy=0x7f32201da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 >> v1:172.21.15.203:6789/0 conn(0x7f32200a98f0 legacy=0x7f3220137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 1 monclient: found mon.a 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220138040 con 0x7f32201430c0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 10 monclient: _finish_auth 0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f321cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f32269d6640 5 monclient: authenticate success, global_id 4530 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f32269d6640 10 monclient: _renew_subs 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f32269d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:48.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.150+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32201382f0 con 0x7f32201430c0 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.151+0000 7f32269d6640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.151+0000 7f32269d6640 10 monclient: _renew_subs 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.151+0000 7f32269d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.151+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32201e2fb0 con 0x7f32201430c0 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.151+0000 7f32269d6640 1 librados: init done 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.152+0000 7f32269d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.152+0000 7f32269d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:48.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.152+0000 7f32269d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:48.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.152+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32200b3ef0 con 0x7f32201430c0 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3218002020 con 0x7f32201430c0 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f3218003dc0 con 0x7f32201430c0 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:48.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321cff9640 10 monclient: dump: 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:48.442 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.440+0000 7f321e7fc640 4 set_mon_vals no callback set 2024-03-25T17:35:48.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.441+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f3218019d90 con 0x7f32201430c0 2024-03-25T17:35:48.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.442+0000 7f321ffff640 10 monclient: get_auth_request con 0x7f320c06df20 auth_method 0 2024-03-25T17:35:48.443 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.442+0000 7f321cff9640 10 client.4530.objecter ms_handle_connect 0x7f320c06df20 2024-03-25T17:35:48.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.446+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f3218053920 con 0x7f32201430c0 2024-03-25T17:35:48.447 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.446+0000 7f321cff9640 10 client.4530.objecter ms_dispatch 0x7f3220000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.446+0000 7f321cff9640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.446+0000 7f321cff9640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.447+0000 7f321cff9640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.447+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f321804f260 con 0x7f32201430c0 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.447+0000 7f321cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:48.448 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.447+0000 7f321cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:48.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.694+0000 7f32269d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:48.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.694+0000 7f32269d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:48.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.694+0000 7f32269d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:48.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.694+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32200627b0 con 0x7f32201430c0 2024-03-25T17:35:48.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.725+0000 7f321cff9640 1 -- 172.21.15.96:0/2038265347 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f3218051360 con 0x7f32201430c0 2024-03-25T17:35:48.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.725+0000 7f321cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:48.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.725+0000 7f321cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:48.727 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:48.727 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 10 librados: watch_flush enter 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 10 librados: watch_flush exit 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 10 client.4530.objecter successfully canceled tick 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 >> v1:172.21.15.203:6800/3611964061 conn(0x7f320c06df20 legacy=0x7f320c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 10 monclient: shutdown 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 >> v1:172.21.15.96:6789/0 conn(0x7f32201430c0 legacy=0x7f32201daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:48.729 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 shutdown_connections 2024-03-25T17:35:48.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 >> 172.21.15.96:0/2038265347 conn(0x7f32200b7ea0 msgr2=0x7f3220074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:48.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 shutdown_connections 2024-03-25T17:35:48.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 -- 172.21.15.96:0/2038265347 wait complete. 2024-03-25T17:35:48.730 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:48.727+0000 7f32269d6640 1 librados: shutdown 2024-03-25T17:35:48.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:55.547 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.663+0000 7f52178eb640 10 monclient: get_monmap_and_config 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.663+0000 7f52178eb640 10 monclient: build_initial_monmap 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.663+0000 7f52178eb640 10 monclient: monmap: 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:35:55.664501+0000 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:35:55.664501+0000 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.664 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:55.665 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:35:55.665 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:35:55.665 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.664+0000 7f52178eb640 1 Processor -- start 2024-03-25T17:35:55.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.664+0000 7f52178eb640 1 -- start start 2024-03-25T17:35:55.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.664+0000 7f52178eb640 10 monclient: init 2024-03-25T17:35:55.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:55.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-c con 0x7f52100a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-a con 0x7f521013f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-b con 0x7f52101430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5210142520 con 0x7f521013f710 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b28a0 con 0x7f52100a98f0 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b2a30 con 0x7f52101430c0 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:55.667 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.665+0000 7f52178eb640 10 monclient(hunting): authenticate will time out at 5713.947165s 2024-03-25T17:35:55.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.672+0000 7f5215687640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f52100a98f0 0x7f52100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33668/0 (socket says 172.21.15.96:33668) 2024-03-25T17:35:55.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.672+0000 7f5215687640 1 -- 172.21.15.96:0/4254199101 learned_addr learned my addr 172.21.15.96:0/4254199101 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:55.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089795445 0 0) 0x7f52100b28a0 con 0x7f52100a98f0 2024-03-25T17:35:55.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:55.681 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:55.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:55.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T17:35:55.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.681+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc003520 con 0x7f52100a98f0 2024-03-25T17:35:55.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1657798657 0 0) 0x7f51fc003520 con 0x7f52100a98f0 2024-03-25T17:35:55.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 >> v1:172.21.15.96:6790/0 conn(0x7f52101430c0 legacy=0x7f521014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:55.686 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 >> v1:172.21.15.96:6789/0 conn(0x7f521013f710 legacy=0x7f52101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: _finish_hunting 0 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 monclient: found mon.noname-c 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100b44c0 con 0x7f52100a98f0 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: _finish_auth 0 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: _check_auth_tickets 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f520c001690 con 0x7f52100a98f0 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f5203fff640 10 monclient: dump: 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:55.687 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f52178eb640 5 monclient: authenticate success, global_id 4519 2024-03-25T17:35:55.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.685+0000 7f52178eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f520c001860 con 0x7f52100a98f0 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 1 -- 172.21.15.96:0/4254199101 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f520c001cb0 con 0x7f52100a98f0 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f52178eb640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f52178eb640 4 set_mon_vals no callback set 2024-03-25T17:35:55.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.701+0000 7f5203fff640 10 monclient: dump: 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:55.703 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 10 monclient: shutdown 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 1 -- 172.21.15.96:0/4254199101 >> v1:172.21.15.203:6789/0 conn(0x7f52100a98f0 legacy=0x7f52100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 1 -- 172.21.15.96:0/4254199101 shutdown_connections 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 1 -- 172.21.15.96:0/4254199101 >> 172.21.15.96:0/4254199101 conn(0x7f52100b7ea0 msgr2=0x7f52100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 1 -- 172.21.15.96:0/4254199101 shutdown_connections 2024-03-25T17:35:55.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.702+0000 7f52178eb640 1 -- 172.21.15.96:0/4254199101 wait complete. 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 10 monclient: build_initial_monmap 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 10 monclient: monmap: 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 librados: starting msgr at 2024-03-25T17:35:55.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 librados: starting objecter 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 Processor -- start 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 -- start start 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 librados: setting wanted keys 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 1 librados: calling monclient init 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.703+0000 7f52178eb640 10 monclient: init 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-c con 0x7f52100a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-b con 0x7f521013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient(hunting): picked mon.noname-a con 0x7f52101430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521006fde0 con 0x7f52101430c0 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100700c0 con 0x7f521013f710 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521006e020 con 0x7f52100a98f0 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:55.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.704+0000 7f52178eb640 10 monclient(hunting): authenticate will time out at 5713.986474s 2024-03-25T17:35:55.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.706+0000 7f5215687640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f52100a98f0 0x7f52100707c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:45574/0 (socket says 172.21.15.96:45574) 2024-03-25T17:35:55.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.706+0000 7f5215687640 1 -- 172.21.15.96:0/3112982211 learned_addr learned my addr 172.21.15.96:0/3112982211 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:35:55.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.710+0000 7f5201ffb640 10 client.?.objecter ms_handle_connect 0x7f52100a98f0 2024-03-25T17:35:55.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.710+0000 7f5201ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:55.714 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.713+0000 7f5201ffb640 10 client.?.objecter ms_handle_connect 0x7f521013f710 2024-03-25T17:35:55.714 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.713+0000 7f5201ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:55.719 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.718+0000 7f5201ffb640 10 client.?.objecter ms_handle_connect 0x7f52101430c0 2024-03-25T17:35:55.719 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.718+0000 7f5201ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:55.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567441511 0 0) 0x7f521006fde0 con 0x7f52101430c0 2024-03-25T17:35:55.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:55.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:55.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:55.723 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T17:35:55.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.722+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f4004eb0 con 0x7f52101430c0 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590790097 0 0) 0x7f52100700c0 con 0x7f521013f710 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 10 monclient(hunting): my global_id is 4522 2024-03-25T17:35:55.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.724+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521006fde0 con 0x7f521013f710 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650857648 0 0) 0x7f521006e020 con 0x7f52100a98f0 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T17:35:55.727 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.726+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52100700c0 con 0x7f52100a98f0 2024-03-25T17:35:55.740 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 39912961 0 0) 0x7f52100700c0 con 0x7f52100a98f0 2024-03-25T17:35:55.740 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f521006e020 con 0x7f52100a98f0 2024-03-25T17:35:55.740 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f520c001b70 con 0x7f52100a98f0 2024-03-25T17:35:55.740 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 10 monclient(hunting): dump: 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.739+0000 7f5201ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1950001166 0 0) 0x7f51f4004eb0 con 0x7f52101430c0 2024-03-25T17:35:55.741 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52100700c0 con 0x7f52101430c0 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5208001b90 con 0x7f52101430c0 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.740+0000 7f5201ffb640 10 monclient(hunting): dump: 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:55.742 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427800296 0 0) 0x7f521006fde0 con 0x7f521013f710 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51f4004eb0 con 0x7f521013f710 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5204001810 con 0x7f521013f710 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.745+0000 7f5201ffb640 10 monclient(hunting): dump: 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:55.746 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:55.747 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:55.747 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 705559636 0 0) 0x7f52100700c0 con 0x7f52101430c0 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 >> v1:172.21.15.96:6790/0 conn(0x7f52100a98f0 legacy=0x7f52100707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 >> v1:172.21.15.203:6789/0 conn(0x7f521013f710 legacy=0x7f521014cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 1 monclient: found mon.a 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:55.752 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210070ed0 con 0x7f52101430c0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 10 monclient: _finish_auth 0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f5201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 5 monclient: authenticate success, global_id 4533 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 10 monclient: _renew_subs 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521006e5a0 con 0x7f52101430c0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 10 monclient: _renew_subs 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f521006eff0 con 0x7f52101430c0 2024-03-25T17:35:55.753 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.751+0000 7f52178eb640 1 librados: init done 2024-03-25T17:35:55.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.753+0000 7f52178eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:55.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.753+0000 7f52178eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:55.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.753+0000 7f52178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:55.754 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:55.753+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52100b3ef0 con 0x7f52101430c0 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5208002080 con 0x7f52101430c0 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5208003b10 con 0x7f52101430c0 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f5201ffb640 10 monclient: dump: 2024-03-25T17:35:56.070 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:35:56.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.069+0000 7f52037fe640 4 set_mon_vals no callback set 2024-03-25T17:35:56.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.070+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5208003d30 con 0x7f52101430c0 2024-03-25T17:35:56.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.071+0000 7f5215687640 10 monclient: get_auth_request con 0x7f51f406e0e0 auth_method 0 2024-03-25T17:35:56.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.071+0000 7f5201ffb640 10 client.4533.objecter ms_handle_connect 0x7f51f406e0e0 2024-03-25T17:35:56.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5208053840 con 0x7f52101430c0 2024-03-25T17:35:56.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 10 client.4533.objecter ms_dispatch 0x7f5210000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:56.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:56.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:56.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T17:35:56.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f520804f350 con 0x7f52101430c0 2024-03-25T17:35:56.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:56.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.075+0000 7f5201ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:56.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.321+0000 7f52178eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:56.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.321+0000 7f52178eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:56.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.321+0000 7f52178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:35:56.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.321+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521006e1b0 con 0x7f52101430c0 2024-03-25T17:35:56.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.345+0000 7f5201ffb640 1 -- 172.21.15.96:0/3112982211 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5208051280 con 0x7f52101430c0 2024-03-25T17:35:56.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.345+0000 7f5201ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:56.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.345+0000 7f5201ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:56.347 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:35:56.347 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:56.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 10 librados: watch_flush enter 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 10 librados: watch_flush exit 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 10 client.4533.objecter successfully canceled tick 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 >> v1:172.21.15.203:6800/3611964061 conn(0x7f51f406e0e0 legacy=0x7f51f40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 10 monclient: shutdown 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 >> v1:172.21.15.96:6789/0 conn(0x7f52101430c0 legacy=0x7f52101df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 shutdown_connections 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.348+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 >> 172.21.15.96:0/3112982211 conn(0x7f52100b7ea0 msgr2=0x7f5210074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.349+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 shutdown_connections 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.349+0000 7f52178eb640 1 -- 172.21.15.96:0/3112982211 wait complete. 2024-03-25T17:35:56.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:35:56.349+0000 7f52178eb640 1 librados: shutdown 2024-03-25T17:35:56.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:03.168 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:03.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.284+0000 7ff2d7108640 10 monclient: get_monmap_and_config 2024-03-25T17:36:03.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.284+0000 7ff2d7108640 10 monclient: build_initial_monmap 2024-03-25T17:36:03.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.284+0000 7ff2d7108640 10 monclient: monmap: 2024-03-25T17:36:03.285 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:03.285845+0000 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:03.285845+0000 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:03.286 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.285+0000 7ff2d7108640 1 Processor -- start 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.285+0000 7ff2d7108640 1 -- start start 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.285+0000 7ff2d7108640 10 monclient: init 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-a con 0x7ff2d00a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-b con 0x7ff2d01430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d0142520 con 0x7ff2d00a98f0 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00b28a0 con 0x7ff2d013f710 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00b2a30 con 0x7ff2d01430c0 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:03.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.286+0000 7ff2d7108640 10 monclient(hunting): authenticate will time out at 5721.568324s 2024-03-25T17:36:03.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.291+0000 7ff2d56a5640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ff2d01430c0 0x7ff2d014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:50316/0 (socket says 172.21.15.96:50316) 2024-03-25T17:36:03.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.291+0000 7ff2d56a5640 1 -- 172.21.15.96:0/1892367318 learned_addr learned my addr 172.21.15.96:0/1892367318 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:03.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711093248 0 0) 0x7ff2d00b2a30 con 0x7ff2d01430c0 2024-03-25T17:36:03.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T17:36:03.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.304+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c0003520 con 0x7ff2d01430c0 2024-03-25T17:36:03.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256719025 0 0) 0x7ff2d0142520 con 0x7ff2d00a98f0 2024-03-25T17:36:03.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 10 monclient(hunting): my global_id is 4536 2024-03-25T17:36:03.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.307+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d00b2a30 con 0x7ff2d00a98f0 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132002522 0 0) 0x7ff2d00b28a0 con 0x7ff2d013f710 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T17:36:03.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.309+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d0142520 con 0x7ff2d013f710 2024-03-25T17:36:03.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706088258 0 0) 0x7ff2d0142520 con 0x7ff2d013f710 2024-03-25T17:36:03.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 >> v1:172.21.15.96:6790/0 conn(0x7ff2d01430c0 legacy=0x7ff2d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 >> v1:172.21.15.96:6789/0 conn(0x7ff2d00a98f0 legacy=0x7ff2d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 monclient: found mon.noname-c 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00b44c0 con 0x7ff2d013f710 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff2bc0015b0 con 0x7ff2d013f710 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2cf7fe640 10 monclient: dump: 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:03.324 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2d7108640 5 monclient: authenticate success, global_id 4525 2024-03-25T17:36:03.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.322+0000 7ff2d7108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:03.335 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2bc001710 con 0x7ff2d013f710 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 1 -- 172.21.15.96:0/1892367318 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff2bc001b60 con 0x7ff2d013f710 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2d7108640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2d7108640 4 set_mon_vals no callback set 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.335+0000 7ff2cf7fe640 10 monclient: dump: 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.336 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:03.337 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:03.337 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:03.337 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 10 monclient: shutdown 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 1 -- 172.21.15.96:0/1892367318 >> v1:172.21.15.203:6789/0 conn(0x7ff2d013f710 legacy=0x7ff2d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 1 -- 172.21.15.96:0/1892367318 shutdown_connections 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 1 -- 172.21.15.96:0/1892367318 >> 172.21.15.96:0/1892367318 conn(0x7ff2d00b7ea0 msgr2=0x7ff2d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 1 -- 172.21.15.96:0/1892367318 shutdown_connections 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 1 -- 172.21.15.96:0/1892367318 wait complete. 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 10 monclient: build_initial_monmap 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.336+0000 7ff2d7108640 10 monclient: monmap: 2024-03-25T17:36:03.339 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 librados: starting msgr at 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 librados: starting objecter 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 Processor -- start 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 -- start start 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 librados: setting wanted keys 2024-03-25T17:36:03.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 1 librados: calling monclient init 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.337+0000 7ff2d7108640 10 monclient: init 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-a con 0x7ff2d00a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-b con 0x7ff2d013f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d01430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00abb90 con 0x7ff2d00a98f0 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d01da920 con 0x7ff2d013f710 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d0137790 con 0x7ff2d01430c0 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:03.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.338+0000 7ff2d7108640 10 monclient(hunting): authenticate will time out at 5721.620329s 2024-03-25T17:36:03.347 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.346+0000 7ff2d4ea4640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff2d00a98f0 0x7ff2d01dae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42160/0 (socket says 172.21.15.96:42160) 2024-03-25T17:36:03.347 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.346+0000 7ff2d4ea4640 1 -- 172.21.15.96:0/3210722066 learned_addr learned my addr 172.21.15.96:0/3210722066 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:03.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.349+0000 7ff2cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff2d013f710 2024-03-25T17:36:03.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.349+0000 7ff2cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:03.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.355+0000 7ff2cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff2d01430c0 2024-03-25T17:36:03.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.355+0000 7ff2cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:03.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.355+0000 7ff2cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff2d00a98f0 2024-03-25T17:36:03.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.355+0000 7ff2cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:03.361 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046965559 0 0) 0x7ff2d00abb90 con 0x7ff2d00a98f0 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 10 monclient(hunting): my global_id is 4539 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.360+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b4004eb0 con 0x7ff2d00a98f0 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058928460 0 0) 0x7ff2d0137790 con 0x7ff2d01430c0 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 10 monclient(hunting): my global_id is 4526 2024-03-25T17:36:03.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.361+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d00abb90 con 0x7ff2d01430c0 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447236750 0 0) 0x7ff2d01da920 con 0x7ff2d013f710 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 10 monclient(hunting): my global_id is 4528 2024-03-25T17:36:03.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.362+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d0137790 con 0x7ff2d013f710 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531103831 0 0) 0x7ff2d0137790 con 0x7ff2d013f710 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2d01da920 con 0x7ff2d013f710 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff2bc001a20 con 0x7ff2d013f710 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.369+0000 7ff2cd7fa640 10 monclient(hunting): dump: 2024-03-25T17:36:03.370 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.371 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.370+0000 7ff2cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3606300750 0 0) 0x7ff2b4004eb0 con 0x7ff2d00a98f0 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2d0137790 con 0x7ff2d00a98f0 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff2c4001a10 con 0x7ff2d00a98f0 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:03.377 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.376+0000 7ff2cd7fa640 10 monclient(hunting): dump: 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:03.378 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1464202808 0 0) 0x7ff2d01da920 con 0x7ff2d013f710 2024-03-25T17:36:03.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 >> v1:172.21.15.96:6790/0 conn(0x7ff2d01430c0 legacy=0x7ff2d01e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 >> v1:172.21.15.96:6789/0 conn(0x7ff2d00a98f0 legacy=0x7ff2d01dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:36:03.384 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 1 monclient: found mon.b 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0137920 con 0x7ff2d013f710 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 10 monclient: _finish_auth 0 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 5 monclient: authenticate success, global_id 4528 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 10 monclient: _renew_subs 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d01db550 con 0x7ff2d013f710 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 10 monclient: _renew_subs 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d0138c00 con 0x7ff2d013f710 2024-03-25T17:36:03.385 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.383+0000 7ff2d7108640 1 librados: init done 2024-03-25T17:36:03.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.385+0000 7ff2d7108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:03.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.385+0000 7ff2d7108640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:03.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.385+0000 7ff2d7108640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.385+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d00627b0 con 0x7ff2d013f710 2024-03-25T17:36:03.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2bc001b80 con 0x7ff2d013f710 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ff2bc003a40 con 0x7ff2d013f710 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2cd7fa640 10 monclient: dump: 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:03.421 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.420+0000 7ff2ceffd640 4 set_mon_vals no callback set 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.421+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ff2bc019a10 con 0x7ff2d013f710 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.421+0000 7ff2d4ea4640 10 monclient: get_auth_request con 0x7ff2b406df10 auth_method 0 2024-03-25T17:36:03.422 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.422+0000 7ff2cd7fa640 10 client.4528.objecter ms_handle_connect 0x7ff2b406df10 2024-03-25T17:36:03.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.687+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ff2bc0535e0 con 0x7ff2d013f710 2024-03-25T17:36:03.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.687+0000 7ff2cd7fa640 10 client.4528.objecter ms_dispatch 0x7ff2d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:03.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.687+0000 7ff2cd7fa640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:03.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.687+0000 7ff2cd7fa640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:03.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.687+0000 7ff2cd7fa640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T17:36:03.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.688+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2bc04f0f0 con 0x7ff2d013f710 2024-03-25T17:36:03.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.688+0000 7ff2cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:03.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.688+0000 7ff2cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:03.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.929+0000 7ff2d7108640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:03.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.929+0000 7ff2d7108640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:03.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.929+0000 7ff2d7108640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:03.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.929+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2d01daab0 con 0x7ff2d013f710 2024-03-25T17:36:03.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.948+0000 7ff2cd7fa640 1 -- 172.21.15.96:0/3210722066 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ff2bc051020 con 0x7ff2d013f710 2024-03-25T17:36:03.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.948+0000 7ff2cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:03.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.948+0000 7ff2cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:03.949 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:03.949 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:03.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 10 librados: watch_flush enter 2024-03-25T17:36:03.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 10 librados: watch_flush exit 2024-03-25T17:36:03.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T17:36:03.952 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 10 client.4528.objecter successfully canceled tick 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 >> v1:172.21.15.203:6800/3611964061 conn(0x7ff2b406df10 legacy=0x7ff2b40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 10 monclient: shutdown 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 >> v1:172.21.15.203:6789/0 conn(0x7ff2d013f710 legacy=0x7ff2d0138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 shutdown_connections 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 >> 172.21.15.96:0/3210722066 conn(0x7ff2d00b7ea0 msgr2=0x7ff2d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.951+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 shutdown_connections 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.952+0000 7ff2d7108640 1 -- 172.21.15.96:0/3210722066 wait complete. 2024-03-25T17:36:03.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:03.952+0000 7ff2d7108640 1 librados: shutdown 2024-03-25T17:36:03.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:06.158 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:30.384 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.162 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:30.388 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.193 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:30.419 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.213 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:30.428 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.216 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:30.431 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.222 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:30.448 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.231 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:30.458 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.252 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:30.467 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:06.273 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:30.499 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.769 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.882+0000 7f8768b9a640 10 monclient: get_monmap_and_config 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.882+0000 7f8768b9a640 10 monclient: build_initial_monmap 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.882+0000 7f8768b9a640 10 monclient: monmap: 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:10.883735+0000 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:10.883735+0000 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:10.883 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:10.884 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:10.884 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:10.884 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:10.884 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 1 Processor -- start 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 1 -- start start 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient: init 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-c con 0x7f87640a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-b con 0x7f876413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-a con 0x7f87641430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8764142520 con 0x7f87641430c0 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.883+0000 7f8768b9a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87640b28a0 con 0x7f87640a98f0 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.884+0000 7f8768b9a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87640b2a30 con 0x7f876413f710 2024-03-25T17:36:10.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.884+0000 7f8768b9a640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:10.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.884+0000 7f8768b9a640 10 monclient(hunting): authenticate will time out at 5729.165959s 2024-03-25T17:36:10.891 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.890+0000 7f8761d9b640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f876413f710 0x7f87641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:36900/0 (socket says 172.21.15.96:36900) 2024-03-25T17:36:10.891 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.890+0000 7f8761d9b640 1 -- 172.21.15.96:0/612438945 learned_addr learned my addr 172.21.15.96:0/612438945 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:10.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.901+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707866636 0 0) 0x7f87640b2a30 con 0x7f876413f710 2024-03-25T17:36:10.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.902+0000 7f876159a640 10 monclient(hunting): _negotiate 2024-03-25T17:36:10.902 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.902+0000 7f876159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:10.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.902+0000 7f876159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:10.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.902+0000 7f876159a640 10 monclient(hunting): my global_id is 4529 2024-03-25T17:36:10.903 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.902+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c003520 con 0x7f876413f710 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672506467 0 0) 0x7f874c003520 con 0x7f876413f710 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 >> v1:172.21.15.203:6789/0 conn(0x7f87640a98f0 legacy=0x7f87640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 >> v1:172.21.15.96:6789/0 conn(0x7f87641430c0 legacy=0x7f876414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: _finish_hunting 0 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 monclient: found mon.noname-b 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:10.916 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640b44c0 con 0x7f876413f710 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: _finish_auth 0 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: _check_auth_tickets 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8758001690 con 0x7f876413f710 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f876159a640 10 monclient: dump: 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:10.917 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:10.918 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:10.918 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:10.918 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:10.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f8768b9a640 5 monclient: authenticate success, global_id 4529 2024-03-25T17:36:10.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.915+0000 7f8768b9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87580017f0 con 0x7f876413f710 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 1 -- 172.21.15.96:0/612438945 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8758001c40 con 0x7f876413f710 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f876159a640 10 monclient: dump: 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:10.929 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f8768b9a640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:10.930 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.928+0000 7f8768b9a640 4 set_mon_vals no callback set 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 10 monclient: shutdown 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 1 -- 172.21.15.96:0/612438945 >> v1:172.21.15.96:6790/0 conn(0x7f876413f710 legacy=0x7f87641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 1 -- 172.21.15.96:0/612438945 shutdown_connections 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 1 -- 172.21.15.96:0/612438945 >> 172.21.15.96:0/612438945 conn(0x7f87640b7ea0 msgr2=0x7f87640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 1 -- 172.21.15.96:0/612438945 shutdown_connections 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.929+0000 7f8768b9a640 1 -- 172.21.15.96:0/612438945 wait complete. 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.930+0000 7f8768b9a640 10 monclient: build_initial_monmap 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.930+0000 7f8768b9a640 10 monclient: monmap: 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:10.932 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 librados: starting msgr at 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 librados: starting objecter 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 Processor -- start 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 -- start start 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 librados: setting wanted keys 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 1 librados: calling monclient init 2024-03-25T17:36:10.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.931+0000 7f8768b9a640 10 monclient: init 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-a con 0x7f87640a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-b con 0x7f876413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient(hunting): picked mon.noname-c con 0x7f87641430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8764135730 con 0x7f87640a98f0 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87641358c0 con 0x7f876413f710 2024-03-25T17:36:10.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8764135a50 con 0x7f87641430c0 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.932+0000 7f8768b9a640 10 monclient(hunting): authenticate will time out at 5729.214585s 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.934+0000 7f8762d9d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f87641430c0 0x7f87641d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:36944/0 (socket says 172.21.15.96:36944) 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.934+0000 7f8762d9d640 1 -- 172.21.15.96:0/203356548 learned_addr learned my addr 172.21.15.96:0/203356548 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.934+0000 7f8752ffd640 10 client.?.objecter ms_handle_connect 0x7f87641430c0 2024-03-25T17:36:10.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.934+0000 7f8752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:10.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.937+0000 7f8752ffd640 10 client.?.objecter ms_handle_connect 0x7f876413f710 2024-03-25T17:36:10.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.937+0000 7f8752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:10.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.991+0000 7f8752ffd640 10 client.?.objecter ms_handle_connect 0x7f87640a98f0 2024-03-25T17:36:10.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:10.991+0000 7f8752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:11.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719119892 0 0) 0x7f8764135730 con 0x7f87640a98f0 2024-03-25T17:36:11.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:11.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:11.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:11.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T17:36:11.007 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.005+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8744004eb0 con 0x7f87640a98f0 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625098200 0 0) 0x7f8744004eb0 con 0x7f87640a98f0 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8764135730 con 0x7f87640a98f0 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f8754001930 con 0x7f87640a98f0 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 10 monclient(hunting): dump: 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:11.014 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:11.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.013+0000 7f8752ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4262158309 0 0) 0x7f8764135730 con 0x7f87640a98f0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 >> v1:172.21.15.96:6790/0 conn(0x7f87641430c0 legacy=0x7f87641d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 >> v1:172.21.15.203:6789/0 conn(0x7f876413f710 legacy=0x7f87641d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 1 monclient: found mon.a 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87641d5000 con 0x7f87640a98f0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 10 monclient: _finish_auth 0 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.019+0000 7f8752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:11.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 5 monclient: authenticate success, global_id 4545 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 10 monclient: _renew_subs 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87641d52b0 con 0x7f87640a98f0 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 10 monclient: _renew_subs 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f876413c900 con 0x7f87640a98f0 2024-03-25T17:36:11.022 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.020+0000 7f8768b9a640 1 librados: init done 2024-03-25T17:36:11.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.021+0000 7f8768b9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:11.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.021+0000 7f8768b9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:11.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.021+0000 7f8768b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:11.023 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.021+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87641371c0 con 0x7f87640a98f0 2024-03-25T17:36:11.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.233+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8754003610 con 0x7f87640a98f0 2024-03-25T17:36:11.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.233+0000 7f8752ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:11.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.233+0000 7f8760d99640 4 set_mon_vals no callback set 2024-03-25T17:36:11.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.266+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f87540039e0 con 0x7f87640a98f0 2024-03-25T17:36:11.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.266+0000 7f8752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:11.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.266+0000 7f8752ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:11.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.266+0000 7f8752ffd640 10 monclient: dump: 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:11.268 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:11.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.267+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f8754003bc0 con 0x7f87640a98f0 2024-03-25T17:36:11.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.268+0000 7f8761d9b640 10 monclient: get_auth_request con 0x7f874406dc90 auth_method 0 2024-03-25T17:36:11.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.268+0000 7f8752ffd640 10 client.4545.objecter ms_handle_connect 0x7f874406dc90 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f8754053670 con 0x7f87640a98f0 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 10 client.4545.objecter ms_dispatch 0x7f8764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87540b08b0 con 0x7f87640a98f0 2024-03-25T17:36:11.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:11.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.275+0000 7f8752ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:11.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.547+0000 7f8768b9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:11.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.547+0000 7f8768b9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:11.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.547+0000 7f8768b9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:11.548 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.547+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87640627b0 con 0x7f87640a98f0 2024-03-25T17:36:11.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.573+0000 7f8752ffd640 1 -- 172.21.15.96:0/203356548 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f875404f180 con 0x7f87640a98f0 2024-03-25T17:36:11.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.573+0000 7f8752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:11.574 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.573+0000 7f8752ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:11.575 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:11.575 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 10 librados: watch_flush enter 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 10 librados: watch_flush exit 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 10 client.4545.objecter successfully canceled tick 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 >> v1:172.21.15.203:6800/3611964061 conn(0x7f874406dc90 legacy=0x7f8744070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:11.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 10 monclient: shutdown 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 >> v1:172.21.15.96:6789/0 conn(0x7f87640a98f0 legacy=0x7f87641dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 shutdown_connections 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 >> 172.21.15.96:0/203356548 conn(0x7f87640b7ea0 msgr2=0x7f876414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 shutdown_connections 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 -- 172.21.15.96:0/203356548 wait complete. 2024-03-25T17:36:11.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:11.576+0000 7f8768b9a640 1 librados: shutdown 2024-03-25T17:36:11.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:18.396 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.516+0000 7fe4aadaf640 10 monclient: get_monmap_and_config 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.516+0000 7fe4aadaf640 10 monclient: build_initial_monmap 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.516+0000 7fe4aadaf640 10 monclient: monmap: 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:18.517897+0000 2024-03-25T17:36:18.517 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:18.517897+0000 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:18.518 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.517+0000 7fe4aadaf640 1 Processor -- start 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.517+0000 7fe4aadaf640 1 -- start start 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.517+0000 7fe4aadaf640 10 monclient: init 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:18.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a41430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a4142520 con 0x7fe4a40a98f0 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40b28a0 con 0x7fe4a413f710 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40b2a30 con 0x7fe4a41430c0 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:18.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.518+0000 7fe4aadaf640 10 monclient(hunting): authenticate will time out at 5736.800406s 2024-03-25T17:36:18.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.536+0000 7fe4a8b4b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe4a40a98f0 0x7fe4a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56342/0 (socket says 172.21.15.96:56342) 2024-03-25T17:36:18.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.536+0000 7fe4a8b4b640 1 -- 172.21.15.96:0/46713865 learned_addr learned my addr 172.21.15.96:0/46713865 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776249236 0 0) 0x7fe4a4142520 con 0x7fe4a40a98f0 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 10 monclient(hunting): my global_id is 4548 2024-03-25T17:36:18.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.551+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe494003520 con 0x7fe4a40a98f0 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513454819 0 0) 0x7fe4a40b28a0 con 0x7fe4a413f710 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 10 monclient(hunting): my global_id is 4534 2024-03-25T17:36:18.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.554+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4142520 con 0x7fe4a413f710 2024-03-25T17:36:18.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005984672 0 0) 0x7fe4a40b2a30 con 0x7fe4a41430c0 2024-03-25T17:36:18.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 10 monclient(hunting): my global_id is 4535 2024-03-25T17:36:18.564 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.562+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a40b28a0 con 0x7fe4a41430c0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106687961 0 0) 0x7fe4a40b28a0 con 0x7fe4a41430c0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 >> v1:172.21.15.203:6789/0 conn(0x7fe4a413f710 legacy=0x7fe4a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 >> v1:172.21.15.96:6789/0 conn(0x7fe4a40a98f0 legacy=0x7fe4a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 monclient: found mon.noname-b 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40b44c0 con 0x7fe4a41430c0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: _finish_auth 0 2024-03-25T17:36:18.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe49c001930 con 0x7fe4a41430c0 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4a37fe640 10 monclient: dump: 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:18.582 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:18.583 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4aadaf640 5 monclient: authenticate success, global_id 4535 2024-03-25T17:36:18.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.580+0000 7fe4aadaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe49c001a90 con 0x7fe4a41430c0 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 1 -- 172.21.15.96:0/46713865 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe49c001ee0 con 0x7fe4a41430c0 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4aadaf640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4aadaf640 4 set_mon_vals no callback set 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:18.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.599+0000 7fe4a37fe640 10 monclient: dump: 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:18.601 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 10 monclient: shutdown 2024-03-25T17:36:18.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:18.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 1 -- 172.21.15.96:0/46713865 >> v1:172.21.15.96:6790/0 conn(0x7fe4a41430c0 legacy=0x7fe4a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:18.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 1 -- 172.21.15.96:0/46713865 shutdown_connections 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 1 -- 172.21.15.96:0/46713865 >> 172.21.15.96:0/46713865 conn(0x7fe4a40b7ea0 msgr2=0x7fe4a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 1 -- 172.21.15.96:0/46713865 shutdown_connections 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.600+0000 7fe4aadaf640 1 -- 172.21.15.96:0/46713865 wait complete. 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.601+0000 7fe4aadaf640 10 monclient: build_initial_monmap 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.601+0000 7fe4aadaf640 10 monclient: monmap: 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:18.604 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.601+0000 7fe4aadaf640 1 librados: starting msgr at 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.601+0000 7fe4aadaf640 1 librados: starting objecter 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.601+0000 7fe4aadaf640 1 Processor -- start 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 -- start start 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 librados: setting wanted keys 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 librados: calling monclient init 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient: init 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a41397e0 con 0x7fe4a41430c0 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40ab9e0 con 0x7fe4a413f710 2024-03-25T17:36:18.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40abcc0 con 0x7fe4a40a98f0 2024-03-25T17:36:18.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:18.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.602+0000 7fe4aadaf640 10 monclient(hunting): authenticate will time out at 5736.884682s 2024-03-25T17:36:18.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.612+0000 7fe4a3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe4a413f710 0x7fe4a40ac150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:58666/0 (socket says 172.21.15.96:58666) 2024-03-25T17:36:18.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.612+0000 7fe4a3fff640 1 -- 172.21.15.96:0/2558578809 learned_addr learned my addr 172.21.15.96:0/2558578809 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:18.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.614+0000 7fe4a17fa640 10 client.?.objecter ms_handle_connect 0x7fe4a413f710 2024-03-25T17:36:18.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.614+0000 7fe4a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:18.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.621+0000 7fe4a17fa640 10 client.?.objecter ms_handle_connect 0x7fe4a41430c0 2024-03-25T17:36:18.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.621+0000 7fe4a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435643756 0 0) 0x7fe4a40ab9e0 con 0x7fe4a413f710 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 monclient(hunting): my global_id is 4537 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe488004eb0 con 0x7fe4a413f710 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 client.?.objecter ms_handle_connect 0x7fe4a40a98f0 2024-03-25T17:36:18.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.624+0000 7fe4a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:18.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.630+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422547086 0 0) 0x7fe4a41397e0 con 0x7fe4a41430c0 2024-03-25T17:36:18.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.631+0000 7fe4a17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.631+0000 7fe4a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.631 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.631+0000 7fe4a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.631+0000 7fe4a17fa640 10 monclient(hunting): my global_id is 4551 2024-03-25T17:36:18.632 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.631+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a40ab9e0 con 0x7fe4a41430c0 2024-03-25T17:36:18.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358034794 0 0) 0x7fe4a40abcc0 con 0x7fe4a40a98f0 2024-03-25T17:36:18.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:18.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:18.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:18.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 10 monclient(hunting): my global_id is 4538 2024-03-25T17:36:18.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.637+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a41397e0 con 0x7fe4a40a98f0 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464219603 0 0) 0x7fe4a40ab9e0 con 0x7fe4a41430c0 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4a40abcc0 con 0x7fe4a41430c0 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe49c001da0 con 0x7fe4a41430c0 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 10 monclient(hunting): dump: 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:18.644 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:18.645 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:18.645 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:18.645 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:18.645 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.643+0000 7fe4a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:18.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3492137142 0 0) 0x7fe4a40abcc0 con 0x7fe4a41430c0 2024-03-25T17:36:18.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 >> v1:172.21.15.96:6790/0 conn(0x7fe4a40a98f0 legacy=0x7fe4a41384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:18.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 >> v1:172.21.15.203:6789/0 conn(0x7fe4a413f710 legacy=0x7fe4a40ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:18.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:36:18.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 1 monclient: found mon.a 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.654+0000 7fe4a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4138d30 con 0x7fe4a41430c0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4a17fa640 10 monclient: _finish_auth 0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4a17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 5 monclient: authenticate success, global_id 4551 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _renew_subs 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a4137970 con 0x7fe4a41430c0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _renew_subs 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:18.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a4136420 con 0x7fe4a41430c0 2024-03-25T17:36:18.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 1 librados: init done 2024-03-25T17:36:18.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:18.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:18.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:18.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.655+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40aa5a0 con 0x7fe4a41430c0 2024-03-25T17:36:18.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe49c003970 con 0x7fe4a41430c0 2024-03-25T17:36:18.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe49c003c80 con 0x7fe4a41430c0 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a17fa640 10 monclient: dump: 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:18.950 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:18.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.948+0000 7fe4a2ffd640 4 set_mon_vals no callback set 2024-03-25T17:36:18.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.949+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fe49c019c50 con 0x7fe4a41430c0 2024-03-25T17:36:18.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.950+0000 7fe4a8b4b640 10 monclient: get_auth_request con 0x7fe48806de20 auth_method 0 2024-03-25T17:36:18.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.950+0000 7fe4a17fa640 10 client.4551.objecter ms_handle_connect 0x7fe48806de20 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fe49c053880 con 0x7fe4a41430c0 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 10 client.4551.objecter ms_dispatch 0x7fe4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe49c04f1c0 con 0x7fe4a41430c0 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:18.957 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:18.956+0000 7fe4a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:19.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.168+0000 7fe4aadaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:19.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.168+0000 7fe4aadaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:19.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.168+0000 7fe4aadaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:19.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.168+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a4138ec0 con 0x7fe4a41430c0 2024-03-25T17:36:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.201+0000 7fe4a17fa640 1 -- 172.21.15.96:0/2558578809 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fe49c0512c0 con 0x7fe4a41430c0 2024-03-25T17:36:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.201+0000 7fe4a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:19.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.201+0000 7fe4a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:19.203 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:19.203 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.204+0000 7fe4aadaf640 10 librados: watch_flush enter 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 10 librados: watch_flush exit 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 10 client.4551.objecter successfully canceled tick 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 >> v1:172.21.15.203:6800/3611964061 conn(0x7fe48806de20 legacy=0x7fe4880702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:19.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 10 monclient: shutdown 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 >> v1:172.21.15.96:6789/0 conn(0x7fe4a41430c0 legacy=0x7fe4a41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 shutdown_connections 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 >> 172.21.15.96:0/2558578809 conn(0x7fe4a40b7ea0 msgr2=0x7fe4a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 shutdown_connections 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 -- 172.21.15.96:0/2558578809 wait complete. 2024-03-25T17:36:19.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:19.205+0000 7fe4aadaf640 1 librados: shutdown 2024-03-25T17:36:19.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:26.024 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.141+0000 7f06fb657640 10 monclient: get_monmap_and_config 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.141+0000 7f06fb657640 10 monclient: build_initial_monmap 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.141+0000 7f06fb657640 10 monclient: monmap: 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:26.141683+0000 2024-03-25T17:36:26.141 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:26.141683+0000 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:26.142 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.142+0000 7f06fb657640 1 Processor -- start 2024-03-25T17:36:26.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.142+0000 7f06fb657640 1 -- start start 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.142+0000 7f06fb657640 10 monclient: init 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.142+0000 7f06fb657640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-b con 0x7f06f40b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-a con 0x7f06f40b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-c con 0x7f06f414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40af940 con 0x7f06f40b1730 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40b0a70 con 0x7f06f414be90 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40b0d50 con 0x7f06f40b4d60 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:26.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.143+0000 7f06fb657640 10 monclient(hunting): authenticate will time out at 5744.424175s 2024-03-25T17:36:26.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.153+0000 7f06f93f3640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f06f40b4d60 0x7f06f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:32904/0 (socket says 172.21.15.96:32904) 2024-03-25T17:36:26.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.153+0000 7f06f93f3640 1 -- 172.21.15.96:0/3267258319 learned_addr learned my addr 172.21.15.96:0/3267258319 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:26.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806134483 0 0) 0x7f06f40b0d50 con 0x7f06f40b4d60 2024-03-25T17:36:26.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:26.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:26.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:26.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 10 monclient(hunting): my global_id is 4541 2024-03-25T17:36:26.158 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.157+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0003520 con 0x7f06f40b4d60 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208515537 0 0) 0x7f06f40b0a70 con 0x7f06f414be90 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 10 monclient(hunting): my global_id is 4540 2024-03-25T17:36:26.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.164+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f40b0d50 con 0x7f06f414be90 2024-03-25T17:36:26.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643686681 0 0) 0x7f06e0003520 con 0x7f06f40b4d60 2024-03-25T17:36:26.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 >> v1:172.21.15.203:6789/0 conn(0x7f06f414be90 legacy=0x7f06f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 >> v1:172.21.15.96:6789/0 conn(0x7f06f40b1730 legacy=0x7f06f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: _finish_hunting 0 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 monclient: found mon.noname-b 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40b1030 con 0x7f06f40b4d60 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: _finish_auth 0 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: _check_auth_tickets 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f06f0001690 con 0x7f06f40b4d60 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:26.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06ebfff640 10 monclient: dump: 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06fb657640 5 monclient: authenticate success, global_id 4541 2024-03-25T17:36:26.167 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.165+0000 7f06fb657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f00017f0 con 0x7f06f40b4d60 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 1 -- 172.21.15.96:0/3267258319 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f06f0001c00 con 0x7f06f40b4d60 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06fb657640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06fb657640 4 set_mon_vals no callback set 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:26.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.178+0000 7f06ebfff640 10 monclient: dump: 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:26.179 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 10 monclient: shutdown 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 1 -- 172.21.15.96:0/3267258319 >> v1:172.21.15.96:6790/0 conn(0x7f06f40b4d60 legacy=0x7f06f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 1 -- 172.21.15.96:0/3267258319 shutdown_connections 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 1 -- 172.21.15.96:0/3267258319 >> 172.21.15.96:0/3267258319 conn(0x7f06f40617a0 msgr2=0x7f06f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 1 -- 172.21.15.96:0/3267258319 shutdown_connections 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.179+0000 7f06fb657640 1 -- 172.21.15.96:0/3267258319 wait complete. 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 10 monclient: build_initial_monmap 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 10 monclient: monmap: 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:26.182 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 librados: starting msgr at 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 librados: starting objecter 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 Processor -- start 2024-03-25T17:36:26.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 -- start start 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 librados: setting wanted keys 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 1 librados: calling monclient init 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.180+0000 7f06fb657640 10 monclient: init 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-c con 0x7f06f40b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-b con 0x7f06f40b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient(hunting): picked mon.noname-a con 0x7f06f414be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f41361e0 con 0x7f06f414be90 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f4145080 con 0x7f06f40b4d60 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f4145210 con 0x7f06f40b1730 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:26.184 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.181+0000 7f06fb657640 10 monclient(hunting): authenticate will time out at 5744.462740s 2024-03-25T17:36:26.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.198+0000 7f06f93f3640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f06f40b1730 0x7f06f412ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:32938/0 (socket says 172.21.15.96:32938) 2024-03-25T17:36:26.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.198+0000 7f06f93f3640 1 -- 172.21.15.96:0/1048654536 learned_addr learned my addr 172.21.15.96:0/1048654536 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:26.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.202+0000 7f06e9ffb640 10 client.?.objecter ms_handle_connect 0x7f06f40b1730 2024-03-25T17:36:26.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.202+0000 7f06e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:26.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.208+0000 7f06e9ffb640 10 client.?.objecter ms_handle_connect 0x7f06f40b4d60 2024-03-25T17:36:26.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.208+0000 7f06e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 client.?.objecter ms_handle_connect 0x7f06f414be90 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791545971 0 0) 0x7f06f4145210 con 0x7f06f40b1730 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:26.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:26.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 10 monclient(hunting): my global_id is 4544 2024-03-25T17:36:26.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.210+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d8004eb0 con 0x7f06f40b1730 2024-03-25T17:36:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984984238 0 0) 0x7f06f4145080 con 0x7f06f40b4d60 2024-03-25T17:36:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:26.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 10 monclient(hunting): my global_id is 4543 2024-03-25T17:36:26.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.221+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f4145210 con 0x7f06f40b4d60 2024-03-25T17:36:26.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894261125 0 0) 0x7f06d8004eb0 con 0x7f06f40b1730 2024-03-25T17:36:26.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f4145080 con 0x7f06f40b1730 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f06f0001c00 con 0x7f06f40b1730 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:26.223 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:26.224 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:26.224 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.222+0000 7f06e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042195366 0 0) 0x7f06f41361e0 con 0x7f06f414be90 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 10 monclient(hunting): my global_id is 4557 2024-03-25T17:36:26.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.235+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d8004eb0 con 0x7f06f414be90 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3799731323 0 0) 0x7f06f4145080 con 0x7f06f40b1730 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 >> v1:172.21.15.203:6789/0 conn(0x7f06f40b4d60 legacy=0x7f06f41d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 >> v1:172.21.15.96:6789/0 conn(0x7f06f414be90 legacy=0x7f06f41e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 1 monclient: found mon.c 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f41306a0 con 0x7f06f40b1730 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:36:26.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 5 monclient: authenticate success, global_id 4544 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 10 monclient: _renew_subs 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f4145440 con 0x7f06f40b1730 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 10 monclient: _renew_subs 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f4137ea0 con 0x7f06f40b1730 2024-03-25T17:36:26.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.237+0000 7f06fb657640 1 librados: init done 2024-03-25T17:36:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.238+0000 7f06fb657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.238+0000 7f06fb657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.238+0000 7f06fb657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.238+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4130890 con 0x7f06f40b1730 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f0004a90 con 0x7f06f40b1730 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f06f0004e20 con 0x7f06f40b1730 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 10 monclient: dump: 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:26.544 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06eb7fe640 4 set_mon_vals no callback set 2024-03-25T17:36:26.545 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.544+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f06f0005040 con 0x7f06f40b1730 2024-03-25T17:36:26.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.545+0000 7f06f8bf2640 10 monclient: get_auth_request con 0x7f06d806de00 auth_method 0 2024-03-25T17:36:26.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.545+0000 7f06e9ffb640 10 client.4544.objecter ms_handle_connect 0x7f06d806de00 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f06f0054950 con 0x7f06f40b1730 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 10 client.4544.objecter ms_dispatch 0x7f06f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.550+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06f0050460 con 0x7f06f40b1730 2024-03-25T17:36:26.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.551+0000 7f06e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:26.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.551+0000 7f06e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:26.610 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:50.836 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.622 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:50.849 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.648 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:07:50.874 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.668 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:50.882 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.674 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:50.901 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.678 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:50.893 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.683 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:50.910 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.703 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:50.918 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.725 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:07:50.951 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:26.783 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.783+0000 7f06fb657640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:26.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.784+0000 7f06fb657640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:26.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.784+0000 7f06fb657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:26.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.784+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f40627b0 con 0x7f06f40b1730 2024-03-25T17:36:26.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.838+0000 7f06e9ffb640 1 -- 172.21.15.96:0/1048654536 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f06f0052390 con 0x7f06f40b1730 2024-03-25T17:36:26.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.838+0000 7f06e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:26.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.838+0000 7f06e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:26.838 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:26.838 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 10 librados: watch_flush enter 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 10 librados: watch_flush exit 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 10 client.4544.objecter successfully canceled tick 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 >> v1:172.21.15.203:6800/3611964061 conn(0x7f06d806de00 legacy=0x7f06d80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 10 monclient: shutdown 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 >> v1:172.21.15.96:6790/0 conn(0x7f06f40b1730 legacy=0x7f06f412ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 shutdown_connections 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 >> 172.21.15.96:0/1048654536 conn(0x7f06f40617a0 msgr2=0x7f06f40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:26.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 shutdown_connections 2024-03-25T17:36:26.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 -- 172.21.15.96:0/1048654536 wait complete. 2024-03-25T17:36:26.842 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:26.840+0000 7f06fb657640 1 librados: shutdown 2024-03-25T17:36:26.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:33.656 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.770+0000 7f37fe631640 10 monclient: get_monmap_and_config 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.770+0000 7f37fe631640 10 monclient: build_initial_monmap 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.770+0000 7f37fe631640 10 monclient: monmap: 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:33.771766+0000 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:33.771766+0000 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.771 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:33.772 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:33.772 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:33.772 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 1 Processor -- start 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 1 -- start start 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 10 monclient: init 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-c con 0x7f37f80b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.771+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-b con 0x7f37f80b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:33.773 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-a con 0x7f37f814be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:33.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80af940 con 0x7f37f814be90 2024-03-25T17:36:33.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80b0a70 con 0x7f37f80b4d60 2024-03-25T17:36:33.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80b0d50 con 0x7f37f80b1730 2024-03-25T17:36:33.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:33.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.772+0000 7f37fe631640 10 monclient(hunting): authenticate will time out at 5752.054024s 2024-03-25T17:36:33.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.783+0000 7f37fcbce640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f37f814be90 0x7f37f814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57140/0 (socket says 172.21.15.96:57140) 2024-03-25T17:36:33.784 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.783+0000 7f37fcbce640 1 -- 172.21.15.96:0/1626223990 learned_addr learned my addr 172.21.15.96:0/1626223990 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771040280 0 0) 0x7f37f80af940 con 0x7f37f814be90 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 10 monclient(hunting): my global_id is 4560 2024-03-25T17:36:33.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.793+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e0003520 con 0x7f37f814be90 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082567413 0 0) 0x7f37f80b0a70 con 0x7f37f80b4d60 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T17:36:33.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.795+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f80af940 con 0x7f37f80b4d60 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765986411 0 0) 0x7f37e0003520 con 0x7f37f814be90 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 >> v1:172.21.15.96:6790/0 conn(0x7f37f80b1730 legacy=0x7f37f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 >> v1:172.21.15.203:6789/0 conn(0x7f37f80b4d60 legacy=0x7f37f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 monclient: found mon.noname-a 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f80b1030 con 0x7f37f814be90 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37f0001a10 con 0x7f37f814be90 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:33.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37f6ffd640 10 monclient: dump: 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37fe631640 5 monclient: authenticate success, global_id 4560 2024-03-25T17:36:33.804 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.801+0000 7f37fe631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:33.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f0001b70 con 0x7f37f814be90 2024-03-25T17:36:33.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:33.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 1 -- 172.21.15.96:0/1626223990 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37f0001f80 con 0x7f37f814be90 2024-03-25T17:36:33.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37fe631640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37fe631640 4 set_mon_vals no callback set 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37f6ffd640 10 monclient: dump: 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:33.815 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37fe631640 10 monclient: shutdown 2024-03-25T17:36:33.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37fe631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.814+0000 7f37fe631640 1 -- 172.21.15.96:0/1626223990 >> v1:172.21.15.96:6789/0 conn(0x7f37f814be90 legacy=0x7f37f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.815+0000 7f37fe631640 1 -- 172.21.15.96:0/1626223990 shutdown_connections 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.815+0000 7f37fe631640 1 -- 172.21.15.96:0/1626223990 >> 172.21.15.96:0/1626223990 conn(0x7f37f80617a0 msgr2=0x7f37f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.815+0000 7f37fe631640 1 -- 172.21.15.96:0/1626223990 shutdown_connections 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.815+0000 7f37fe631640 1 -- 172.21.15.96:0/1626223990 wait complete. 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.816+0000 7f37fe631640 10 monclient: build_initial_monmap 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.816+0000 7f37fe631640 10 monclient: monmap: 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:33.817 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:33.818 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:33.818 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:33.818 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.816+0000 7f37fe631640 1 librados: starting msgr at 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.816+0000 7f37fe631640 1 librados: starting objecter 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.816+0000 7f37fe631640 1 Processor -- start 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 -- start start 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 librados: setting wanted keys 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 librados: calling monclient init 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient: init 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:33.819 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-c con 0x7f37f80b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-a con 0x7f37f80b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient(hunting): picked mon.noname-b con 0x7f37f814be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f81d86f0 con 0x7f37f80b4d60 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f813b4f0 con 0x7f37f814be90 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f81d7eb0 con 0x7f37f80b1730 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:33.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.817+0000 7f37fe631640 10 monclient(hunting): authenticate will time out at 5752.099904s 2024-03-25T17:36:33.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.828+0000 7f37f77fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f37f80b4d60 0x7f37f8136fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57206/0 (socket says 172.21.15.96:57206) 2024-03-25T17:36:33.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.828+0000 7f37f77fe640 1 -- 172.21.15.96:0/219845003 learned_addr learned my addr 172.21.15.96:0/219845003 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:33.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.829+0000 7f37f4ff9640 10 client.?.objecter ms_handle_connect 0x7f37f80b4d60 2024-03-25T17:36:33.830 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.829+0000 7f37f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:33.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.833+0000 7f37f4ff9640 10 client.?.objecter ms_handle_connect 0x7f37f814be90 2024-03-25T17:36:33.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.833+0000 7f37f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:33.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.834+0000 7f37f4ff9640 10 client.?.objecter ms_handle_connect 0x7f37f80b1730 2024-03-25T17:36:33.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.834+0000 7f37f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:33.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639927232 0 0) 0x7f37f80b0d50 con 0x7f37f80b4d60 2024-03-25T17:36:33.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:33.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:33.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:33.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 10 monclient(hunting): my global_id is 4563 2024-03-25T17:36:33.841 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.840+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e4004eb0 con 0x7f37f80b4d60 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379091484 0 0) 0x7f37f813b4f0 con 0x7f37f814be90 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T17:36:33.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.845+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f80b0d50 con 0x7f37f814be90 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3447431315 0 0) 0x7f37e4004eb0 con 0x7f37f80b4d60 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f813b4f0 con 0x7f37f80b4d60 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37ec001890 con 0x7f37f80b4d60 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:33.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 10 monclient(hunting): dump: 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.847+0000 7f37f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6519405 0 0) 0x7f37f81d7eb0 con 0x7f37f80b1730 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 10 monclient(hunting): my global_id is 4550 2024-03-25T17:36:33.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.849+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e4004eb0 con 0x7f37f80b1730 2024-03-25T17:36:33.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.862+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2626776472 0 0) 0x7f37f80b0d50 con 0x7f37f814be90 2024-03-25T17:36:33.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.863+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f81d7eb0 con 0x7f37f814be90 2024-03-25T17:36:33.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.863+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37f0001e80 con 0x7f37f814be90 2024-03-25T17:36:33.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.863+0000 7f37f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:33.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.863+0000 7f37f4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.863+0000 7f37f4ff9640 10 monclient(hunting): dump: 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:33.864 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284598351 0 0) 0x7f37e4004eb0 con 0x7f37f80b1730 2024-03-25T17:36:33.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f80b0d50 con 0x7f37f80b1730 2024-03-25T17:36:33.871 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37e8001930 con 0x7f37f80b1730 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.871+0000 7f37f4ff9640 10 monclient(hunting): dump: 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:33.872 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3032566440 0 0) 0x7f37f813b4f0 con 0x7f37f80b4d60 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 >> v1:172.21.15.96:6790/0 conn(0x7f37f80b1730 legacy=0x7f37f81d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 >> v1:172.21.15.203:6789/0 conn(0x7f37f814be90 legacy=0x7f37f81e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 1 monclient: found mon.a 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f81d8040 con 0x7f37f80b4d60 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 10 monclient: _finish_auth 0 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:33.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 5 monclient: authenticate success, global_id 4563 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 10 monclient: _renew_subs 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f813ba70 con 0x7f37f80b4d60 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 10 monclient: _renew_subs 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f81378d0 con 0x7f37f80b4d60 2024-03-25T17:36:33.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.879+0000 7f37fe631640 1 librados: init done 2024-03-25T17:36:33.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.881+0000 7f37fe631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:33.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.881+0000 7f37fe631640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:33.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.881+0000 7f37fe631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:33.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:33.881+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f80627b0 con 0x7f37f80b4d60 2024-03-25T17:36:34.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ec003610 con 0x7f37f80b4d60 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f37ec001d80 con 0x7f37f80b4d60 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f4ff9640 10 monclient: dump: 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:34.180 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.179+0000 7f37f67fc640 4 set_mon_vals no callback set 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.180+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f37ec003a20 con 0x7f37f80b4d60 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.180+0000 7f37f7fff640 10 monclient: get_auth_request con 0x7f37e406e160 auth_method 0 2024-03-25T17:36:34.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.181+0000 7f37f4ff9640 10 client.4563.objecter ms_handle_connect 0x7f37e406e160 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f37ec054660 con 0x7f37f80b4d60 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 10 client.4563.objecter ms_dispatch 0x7f37f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37ec050170 con 0x7f37f80b4d60 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:34.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.190+0000 7f37f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:34.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.440+0000 7f37fe631640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:34.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.440+0000 7f37fe631640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:34.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.440+0000 7f37fe631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:36:34.441 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.440+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f8069f40 con 0x7f37f80b4d60 2024-03-25T17:36:34.478 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.477+0000 7f37f4ff9640 1 -- 172.21.15.96:0/219845003 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f37ec0520a0 con 0x7f37f80b4d60 2024-03-25T17:36:34.478 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.477+0000 7f37f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:34.478 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.477+0000 7f37f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:34.478 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:34.478 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 10 librados: watch_flush enter 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 10 librados: watch_flush exit 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 10 client.4563.objecter successfully canceled tick 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 >> v1:172.21.15.203:6800/3611964061 conn(0x7f37e406e160 legacy=0x7f37e4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:34.481 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 10 monclient: shutdown 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 >> v1:172.21.15.96:6789/0 conn(0x7f37f80b4d60 legacy=0x7f37f8136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 shutdown_connections 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 >> 172.21.15.96:0/219845003 conn(0x7f37f80617a0 msgr2=0x7f37f813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 shutdown_connections 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 -- 172.21.15.96:0/219845003 wait complete. 2024-03-25T17:36:34.482 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:34.480+0000 7f37fe631640 1 librados: shutdown 2024-03-25T17:36:34.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:41.300 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.417+0000 7f58b39d0640 10 monclient: get_monmap_and_config 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.417+0000 7f58b39d0640 10 monclient: build_initial_monmap 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.417+0000 7f58b39d0640 10 monclient: monmap: 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:41.418683+0000 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:41.418683+0000 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:41.418 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:41.419 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:41.419 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.418+0000 7f58b39d0640 1 Processor -- start 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.418+0000 7f58b39d0640 1 -- start start 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.418+0000 7f58b39d0640 10 monclient: init 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-c con 0x7f58ac0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-a con 0x7f58ac0b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-b con 0x7f58ac14be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac0af940 con 0x7f58ac0b1730 2024-03-25T17:36:41.420 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac0b0a70 con 0x7f58ac0b4d60 2024-03-25T17:36:41.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac0b0d50 con 0x7f58ac14be90 2024-03-25T17:36:41.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:41.421 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.419+0000 7f58b39d0640 10 monclient(hunting): authenticate will time out at 5759.701418s 2024-03-25T17:36:41.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.428+0000 7f58b1f6d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f58ac14be90 0x7f58ac14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:36732/0 (socket says 172.21.15.96:36732) 2024-03-25T17:36:41.429 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.428+0000 7f58b1f6d640 1 -- 172.21.15.96:0/1074639748 learned_addr learned my addr 172.21.15.96:0/1074639748 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228796330 0 0) 0x7f58ac0af940 con 0x7f58ac0b1730 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 10 monclient(hunting): my global_id is 4566 2024-03-25T17:36:41.449 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.448+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5898003520 con 0x7f58ac0b1730 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030199973 0 0) 0x7f58ac0b0d50 con 0x7f58ac14be90 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 10 monclient(hunting): my global_id is 4553 2024-03-25T17:36:41.450 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.449+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac0af940 con 0x7f58ac14be90 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1459280471 0 0) 0x7f5898003520 con 0x7f58ac0b1730 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 >> v1:172.21.15.96:6790/0 conn(0x7f58ac14be90 legacy=0x7f58ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 >> v1:172.21.15.203:6789/0 conn(0x7f58ac0b4d60 legacy=0x7f58ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 monclient: found mon.noname-a 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0b1030 con 0x7f58ac0b1730 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: _finish_auth 0 2024-03-25T17:36:41.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f589c001690 con 0x7f58ac0b1730 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58a3fff640 10 monclient: dump: 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:41.458 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58b39d0640 5 monclient: authenticate success, global_id 4566 2024-03-25T17:36:41.459 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.456+0000 7f58b39d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f589c0017f0 con 0x7f58ac0b1730 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 1 -- 172.21.15.96:0/1074639748 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f589c001c00 con 0x7f58ac0b1730 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58b39d0640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58b39d0640 4 set_mon_vals no callback set 2024-03-25T17:36:41.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.465+0000 7f58a3fff640 10 monclient: dump: 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:41.467 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 10 monclient: shutdown 2024-03-25T17:36:41.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:41.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 1 -- 172.21.15.96:0/1074639748 >> v1:172.21.15.96:6789/0 conn(0x7f58ac0b1730 legacy=0x7f58ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:41.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 1 -- 172.21.15.96:0/1074639748 shutdown_connections 2024-03-25T17:36:41.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 1 -- 172.21.15.96:0/1074639748 >> 172.21.15.96:0/1074639748 conn(0x7f58ac0617a0 msgr2=0x7f58ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.466+0000 7f58b39d0640 1 -- 172.21.15.96:0/1074639748 shutdown_connections 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.467+0000 7f58b39d0640 1 -- 172.21.15.96:0/1074639748 wait complete. 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.467+0000 7f58b39d0640 10 monclient: build_initial_monmap 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.467+0000 7f58b39d0640 10 monclient: monmap: 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.469 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 librados: starting msgr at 2024-03-25T17:36:41.470 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 librados: starting objecter 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 Processor -- start 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 -- start start 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 librados: setting wanted keys 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 1 librados: calling monclient init 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 10 monclient: init 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.468+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-c con 0x7f58ac0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-a con 0x7f58ac0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 10 monclient(hunting): picked mon.noname-b con 0x7f58ac14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac12d790 con 0x7f58ac0b4d60 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac136770 con 0x7f58ac14be90 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac136a50 con 0x7f58ac0b1730 2024-03-25T17:36:41.471 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:41.472 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.469+0000 7f58b39d0640 10 monclient(hunting): authenticate will time out at 5759.751025s 2024-03-25T17:36:41.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.474+0000 7f58b176c640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f58ac0b1730 0x7f58ac137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:36792/0 (socket says 172.21.15.96:36792) 2024-03-25T17:36:41.475 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.474+0000 7f58b176c640 1 -- 172.21.15.96:0/972068794 learned_addr learned my addr 172.21.15.96:0/972068794 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:41.480 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.479+0000 7f58a1ffb640 10 client.?.objecter ms_handle_connect 0x7f58ac0b1730 2024-03-25T17:36:41.480 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.479+0000 7f58a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263369943 0 0) 0x7f58ac136a50 con 0x7f58ac0b1730 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T17:36:41.491 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.490+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5890004eb0 con 0x7f58ac0b1730 2024-03-25T17:36:41.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.491+0000 7f58a1ffb640 10 client.?.objecter ms_handle_connect 0x7f58ac0b4d60 2024-03-25T17:36:41.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.491+0000 7f58a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:41.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.496+0000 7f58a1ffb640 10 client.?.objecter ms_handle_connect 0x7f58ac14be90 2024-03-25T17:36:41.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.496+0000 7f58a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637775752 0 0) 0x7f58ac136770 con 0x7f58ac14be90 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 10 monclient(hunting): my global_id is 4552 2024-03-25T17:36:41.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.511+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac136a50 con 0x7f58ac14be90 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301883695 0 0) 0x7f58ac12d790 con 0x7f58ac0b4d60 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T17:36:41.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.513+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac136770 con 0x7f58ac0b4d60 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 73981825 0 0) 0x7f5890004eb0 con 0x7f58ac0b1730 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58ac12d790 con 0x7f58ac0b1730 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f58a8001a30 con 0x7f58ac0b1730 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:41.519 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.518+0000 7f58a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.527+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3871611164 0 0) 0x7f58ac12d790 con 0x7f58ac0b1730 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 >> v1:172.21.15.203:6789/0 conn(0x7f58ac14be90 legacy=0x7f58ac12d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 >> v1:172.21.15.96:6789/0 conn(0x7f58ac0b4d60 legacy=0x7f58ac12c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 1 monclient: found mon.c 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac136d30 con 0x7f58ac0b1730 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:36:41.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 5 monclient: authenticate success, global_id 4556 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 10 monclient: _renew_subs 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac13e370 con 0x7f58ac0b1730 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 10 monclient: _renew_subs 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac13dfa0 con 0x7f58ac0b1730 2024-03-25T17:36:41.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.528+0000 7f58b39d0640 1 librados: init done 2024-03-25T17:36:41.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.529+0000 7f58b39d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:41.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.529+0000 7f58b39d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:41.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.529+0000 7f58b39d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:41.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.529+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac1401f0 con 0x7f58ac0b1730 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58a8001c70 con 0x7f58ac0b1730 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f58a8003a60 con 0x7f58ac0b1730 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 10 monclient: dump: 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:41.823 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a37fe640 4 set_mon_vals no callback set 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.822+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f58a8003c80 con 0x7f58ac0b1730 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.823+0000 7f58b0f6b640 10 monclient: get_auth_request con 0x7f589006de20 auth_method 0 2024-03-25T17:36:41.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.823+0000 7f58a1ffb640 10 client.4556.objecter ms_handle_connect 0x7f589006de20 2024-03-25T17:36:41.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f58a8054820 con 0x7f58ac0b1730 2024-03-25T17:36:41.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 10 client.4556.objecter ms_dispatch 0x7f58ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:41.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:41.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:41.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T17:36:41.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58a8050330 con 0x7f58ac0b1730 2024-03-25T17:36:41.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:41.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:41.832+0000 7f58a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:42.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.091+0000 7f58b39d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:42.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.091+0000 7f58b39d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:42.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.091+0000 7f58b39d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:42.092 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.091+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58ac0afbd0 con 0x7f58ac0b1730 2024-03-25T17:36:42.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.139+0000 7f58a1ffb640 1 -- 172.21.15.96:0/972068794 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f58a8052260 con 0x7f58ac0b1730 2024-03-25T17:36:42.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.139+0000 7f58a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:42.140 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.139+0000 7f58a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:42.140 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:42.140 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:42.142 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 10 librados: watch_flush enter 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 10 librados: watch_flush exit 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 10 client.4556.objecter successfully canceled tick 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 >> v1:172.21.15.203:6800/3611964061 conn(0x7f589006de20 legacy=0x7f58900702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 10 monclient: shutdown 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.141+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 >> v1:172.21.15.96:6790/0 conn(0x7f58ac0b1730 legacy=0x7f58ac137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.142+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 shutdown_connections 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.142+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 >> 172.21.15.96:0/972068794 conn(0x7f58ac0617a0 msgr2=0x7f58ac13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.142+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 shutdown_connections 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.142+0000 7f58b39d0640 1 -- 172.21.15.96:0/972068794 wait complete. 2024-03-25T17:36:42.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:42.142+0000 7f58b39d0640 1 librados: shutdown 2024-03-25T17:36:42.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:46.418 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:10.644 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.470 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:10.696 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.511 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:10.738 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.514 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:10.729 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.524 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:10.739 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.534 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:10.760 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.538 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:10.764 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.548 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:10.763 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:46.561 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:10.787 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:48.960 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.078+0000 7f15228ba640 10 monclient: get_monmap_and_config 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.078+0000 7f15228ba640 10 monclient: build_initial_monmap 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.078+0000 7f15228ba640 10 monclient: monmap: 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:49.079660+0000 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:49.079660+0000 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:49.079 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:49.080 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:49.080 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.079+0000 7f15228ba640 1 Processor -- start 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.079+0000 7f15228ba640 1 -- start start 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.079+0000 7f15228ba640 10 monclient: init 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-a con 0x7f151c0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-b con 0x7f151c13f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-c con 0x7f151c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:49.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c142520 con 0x7f151c0a98f0 2024-03-25T17:36:49.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b28a0 con 0x7f151c1430c0 2024-03-25T17:36:49.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b2a30 con 0x7f151c13f710 2024-03-25T17:36:49.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:49.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.080+0000 7f15228ba640 10 monclient(hunting): authenticate will time out at 5767.362321s 2024-03-25T17:36:49.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.085+0000 7f151b7fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f151c13f710 0x7f151c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:33168/0 (socket says 172.21.15.96:33168) 2024-03-25T17:36:49.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.085+0000 7f151b7fe640 1 -- 172.21.15.96:0/2362256947 learned_addr learned my addr 172.21.15.96:0/2362256947 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711573137 0 0) 0x7f151c0b2a30 con 0x7f151c13f710 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 10 monclient(hunting): my global_id is 4559 2024-03-25T17:36:49.110 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.109+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1504003520 con 0x7f151c13f710 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043137266 0 0) 0x7f151c0b28a0 con 0x7f151c1430c0 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 10 monclient(hunting): my global_id is 4555 2024-03-25T17:36:49.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.112+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c0b2a30 con 0x7f151c1430c0 2024-03-25T17:36:49.121 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799454207 0 0) 0x7f151c0b2a30 con 0x7f151c1430c0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 >> v1:172.21.15.96:6790/0 conn(0x7f151c13f710 legacy=0x7f151c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 >> v1:172.21.15.96:6789/0 conn(0x7f151c0a98f0 legacy=0x7f151c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 10 monclient: _finish_hunting 0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 1 monclient: found mon.noname-c 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.120+0000 7f151affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0b44c0 con 0x7f151c1430c0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: _finish_auth 0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: _check_auth_tickets 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f15228ba640 5 monclient: authenticate success, global_id 4555 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f15228ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f15140019f0 con 0x7f151c1430c0 2024-03-25T17:36:49.122 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f151affd640 10 monclient: dump: 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.123 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.121+0000 7f15228ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1514001b50 con 0x7f151c1430c0 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 1 -- 172.21.15.96:0/2362256947 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1514001f20 con 0x7f151c1430c0 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f15228ba640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f15228ba640 4 set_mon_vals no callback set 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.131+0000 7f151affd640 10 monclient: dump: 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:49.132 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:49.133 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 10 monclient: shutdown 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 1 -- 172.21.15.96:0/2362256947 >> v1:172.21.15.203:6789/0 conn(0x7f151c1430c0 legacy=0x7f151c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 1 -- 172.21.15.96:0/2362256947 shutdown_connections 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 1 -- 172.21.15.96:0/2362256947 >> 172.21.15.96:0/2362256947 conn(0x7f151c0b7ea0 msgr2=0x7f151c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:49.134 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 1 -- 172.21.15.96:0/2362256947 shutdown_connections 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.132+0000 7f15228ba640 1 -- 172.21.15.96:0/2362256947 wait complete. 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.133+0000 7f15228ba640 10 monclient: build_initial_monmap 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.133+0000 7f15228ba640 10 monclient: monmap: 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.133+0000 7f15228ba640 1 librados: starting msgr at 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.133+0000 7f15228ba640 1 librados: starting objecter 2024-03-25T17:36:49.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.133+0000 7f15228ba640 1 Processor -- start 2024-03-25T17:36:49.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 -- start start 2024-03-25T17:36:49.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 librados: setting wanted keys 2024-03-25T17:36:49.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 librados: calling monclient init 2024-03-25T17:36:49.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient: init 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-c con 0x7f151c0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-b con 0x7f151c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient(hunting): picked mon.noname-a con 0x7f151c1430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b01c0 con 0x7f151c1430c0 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b04a0 con 0x7f151c13f710 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0ae9e0 con 0x7f151c0a98f0 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:49.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.134+0000 7f15228ba640 10 monclient(hunting): authenticate will time out at 5767.416790s 2024-03-25T17:36:49.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.138+0000 7f151b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f151c13f710 0x7f151c0b0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:40538/0 (socket says 172.21.15.96:40538) 2024-03-25T17:36:49.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.138+0000 7f151b7fe640 1 -- 172.21.15.96:0/2745717969 learned_addr learned my addr 172.21.15.96:0/2745717969 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:49.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.142+0000 7f1518ff9640 10 client.?.objecter ms_handle_connect 0x7f151c13f710 2024-03-25T17:36:49.143 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.142+0000 7f1518ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588886416 0 0) 0x7f151c0b04a0 con 0x7f151c13f710 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 10 monclient(hunting): my global_id is 4558 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.151+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1508004eb0 con 0x7f151c13f710 2024-03-25T17:36:49.152 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.152+0000 7f1518ff9640 10 client.?.objecter ms_handle_connect 0x7f151c1430c0 2024-03-25T17:36:49.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.152+0000 7f1518ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:49.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.157+0000 7f1518ff9640 10 client.?.objecter ms_handle_connect 0x7f151c0a98f0 2024-03-25T17:36:49.157 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.157+0000 7f1518ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:49.160 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246264208 0 0) 0x7f151c0b01c0 con 0x7f151c1430c0 2024-03-25T17:36:49.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:49.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:49.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:49.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 10 monclient(hunting): my global_id is 4575 2024-03-25T17:36:49.161 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.160+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c0b04a0 con 0x7f151c1430c0 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782788591 0 0) 0x7f1508004eb0 con 0x7f151c13f710 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f151c0b01c0 con 0x7f151c13f710 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1510001990 con 0x7f151c13f710 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 10 monclient(hunting): dump: 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.164 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.165 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.163+0000 7f1518ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776561975 0 0) 0x7f151c0b01c0 con 0x7f151c13f710 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 >> v1:172.21.15.96:6790/0 conn(0x7f151c0a98f0 legacy=0x7f151c0aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 >> v1:172.21.15.96:6789/0 conn(0x7f151c1430c0 legacy=0x7f151c1e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 1 monclient: found mon.b 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0af650 con 0x7f151c13f710 2024-03-25T17:36:49.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 10 monclient: _finish_auth 0 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f1518ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 5 monclient: authenticate success, global_id 4558 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 10 monclient: _renew_subs 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c0b15d0 con 0x7f151c13f710 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 10 monclient: _renew_subs 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c0ae750 con 0x7f151c13f710 2024-03-25T17:36:49.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.169+0000 7f15228ba640 1 librados: init done 2024-03-25T17:36:49.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.170+0000 7f15228ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:49.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.170+0000 7f15228ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:49.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.170+0000 7f15228ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.170+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c0b3ef0 con 0x7f151c13f710 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15100047e0 con 0x7f151c13f710 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1510004bb0 con 0x7f151c13f710 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f1518ff9640 10 monclient: dump: 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:49.456 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:49.457 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.455+0000 7f151a7fc640 4 set_mon_vals no callback set 2024-03-25T17:36:49.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.456+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1510004dd0 con 0x7f151c13f710 2024-03-25T17:36:49.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.457+0000 7f151bfff640 10 monclient: get_auth_request con 0x7f150806dda0 auth_method 0 2024-03-25T17:36:49.458 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.457+0000 7f1518ff9640 10 client.4558.objecter ms_handle_connect 0x7f150806dda0 2024-03-25T17:36:49.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.460+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f1510054890 con 0x7f151c13f710 2024-03-25T17:36:49.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.460+0000 7f1518ff9640 10 client.4558.objecter ms_dispatch 0x7f151c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:49.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.460+0000 7f1518ff9640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:49.461 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.460+0000 7f1518ff9640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:49.462 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.460+0000 7f1518ff9640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T17:36:49.462 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.461+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15100503a0 con 0x7f151c13f710 2024-03-25T17:36:49.462 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.461+0000 7f1518ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:49.462 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.461+0000 7f1518ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:49.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.712+0000 7f15228ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:49.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.712+0000 7f15228ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:49.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.712+0000 7f15228ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:36:49.713 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.712+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f151c0acf30 con 0x7f151c13f710 2024-03-25T17:36:49.733 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.732+0000 7f1518ff9640 1 -- 172.21.15.96:0/2745717969 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f15100522d0 con 0x7f151c13f710 2024-03-25T17:36:49.733 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.732+0000 7f1518ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:49.733 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.732+0000 7f1518ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:49.733 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:49.734 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:49.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 10 librados: watch_flush enter 2024-03-25T17:36:49.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 10 librados: watch_flush exit 2024-03-25T17:36:49.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T17:36:49.736 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 10 client.4558.objecter successfully canceled tick 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 >> v1:172.21.15.203:6800/3611964061 conn(0x7f150806dda0 legacy=0x7f1508070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 10 monclient: shutdown 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 >> v1:172.21.15.203:6789/0 conn(0x7f151c13f710 legacy=0x7f151c0b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 shutdown_connections 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 >> 172.21.15.96:0/2745717969 conn(0x7f151c0b7ea0 msgr2=0x7f151c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 shutdown_connections 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 -- 172.21.15.96:0/2745717969 wait complete. 2024-03-25T17:36:49.737 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:49.735+0000 7f15228ba640 1 librados: shutdown 2024-03-25T17:36:49.747 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:56.554 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.668+0000 7fa226ad9640 10 monclient: get_monmap_and_config 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.668+0000 7fa226ad9640 10 monclient: build_initial_monmap 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.668+0000 7fa226ad9640 10 monclient: monmap: 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:36:56.669907+0000 2024-03-25T17:36:56.669 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:36:56.669907+0000 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:36:56.670 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:56.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.669+0000 7fa226ad9640 1 Processor -- start 2024-03-25T17:36:56.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.669+0000 7fa226ad9640 1 -- start start 2024-03-25T17:36:56.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.669+0000 7fa226ad9640 10 monclient: init 2024-03-25T17:36:56.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:56.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-c con 0x7fa22006c380 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-a con 0x7fa2200b12d0 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-b con 0x7fa2200aedf0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200b16f0 con 0x7fa2200b12d0 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200adfe0 con 0x7fa22006c380 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200add00 con 0x7fa2200aedf0 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:56.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.670+0000 7fa226ad9640 10 monclient(hunting): authenticate will time out at 5774.952440s 2024-03-25T17:36:56.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.676+0000 7fa225076640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fa2200aedf0 0x7fa22014e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:46798/0 (socket says 172.21.15.96:46798) 2024-03-25T17:36:56.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.676+0000 7fa225076640 1 -- 172.21.15.96:0/535805796 learned_addr learned my addr 172.21.15.96:0/535805796 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321672727 0 0) 0x7fa2200b16f0 con 0x7fa2200b12d0 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T17:36:56.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.687+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208003520 con 0x7fa2200b12d0 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590886105 0 0) 0x7fa2200add00 con 0x7fa2200aedf0 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T17:36:56.697 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.696+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2200b16f0 con 0x7fa2200aedf0 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659361012 0 0) 0x7fa208003520 con 0x7fa2200b12d0 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 >> v1:172.21.15.96:6790/0 conn(0x7fa2200aedf0 legacy=0x7fa22014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 >> v1:172.21.15.203:6789/0 conn(0x7fa22006c380 legacy=0x7fa2200b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: _finish_hunting 0 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 monclient: found mon.noname-a 2024-03-25T17:36:56.698 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200ae2c0 con 0x7fa2200b12d0 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: _finish_auth 0 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: _check_auth_tickets 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa210001690 con 0x7fa2200b12d0 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa226ad9640 5 monclient: authenticate success, global_id 4578 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa226ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa2177fe640 10 monclient: dump: 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:56.699 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:56.700 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.697+0000 7fa226ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:56.702 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2100017f0 con 0x7fa2200b12d0 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 1 -- 172.21.15.96:0/535805796 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa210001b80 con 0x7fa2200b12d0 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa226ad9640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa226ad9640 4 set_mon_vals no callback set 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa2177fe640 10 monclient: dump: 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:56.703 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:56.704 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:56.704 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:56.704 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.702+0000 7fa226ad9640 10 monclient: shutdown 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 1 -- 172.21.15.96:0/535805796 >> v1:172.21.15.96:6789/0 conn(0x7fa2200b12d0 legacy=0x7fa2200ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 1 -- 172.21.15.96:0/535805796 shutdown_connections 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 1 -- 172.21.15.96:0/535805796 >> 172.21.15.96:0/535805796 conn(0x7fa2200b5d10 msgr2=0x7fa2200b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 1 -- 172.21.15.96:0/535805796 shutdown_connections 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.703+0000 7fa226ad9640 1 -- 172.21.15.96:0/535805796 wait complete. 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 10 monclient: build_initial_monmap 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 10 monclient: monmap: 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:56.705 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 librados: starting msgr at 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 librados: starting objecter 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 Processor -- start 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 -- start start 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 librados: setting wanted keys 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 1 librados: calling monclient init 2024-03-25T17:36:56.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.704+0000 7fa226ad9640 10 monclient: init 2024-03-25T17:36:56.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:56.707 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-a con 0x7fa22006c380 addr v1:172.21.15.96:6789/0 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-b con 0x7fa2200aedf0 addr v1:172.21.15.203:6789/0 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient(hunting): picked mon.noname-c con 0x7fa2200b12d0 addr v1:172.21.15.96:6790/0 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22006c8e0 con 0x7fa22006c380 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2201c4320 con 0x7fa2200aedf0 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2201c0ef0 con 0x7fa2200b12d0 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:56.708 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.705+0000 7fa226ad9640 10 monclient(hunting): authenticate will time out at 5774.987932s 2024-03-25T17:36:56.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.711+0000 7fa225076640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fa2200b12d0 0x7fa2201c2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:46832/0 (socket says 172.21.15.96:46832) 2024-03-25T17:36:56.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.711+0000 7fa225076640 1 -- 172.21.15.96:0/783704671 learned_addr learned my addr 172.21.15.96:0/783704671 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:36:56.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.712+0000 7fa2157fa640 10 client.?.objecter ms_handle_connect 0x7fa2200b12d0 2024-03-25T17:36:56.712 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.712+0000 7fa2157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157968623 0 0) 0x7fa2201c0ef0 con 0x7fa2200b12d0 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 10 monclient(hunting): my global_id is 4568 2024-03-25T17:36:56.722 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.721+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c004eb0 con 0x7fa2200b12d0 2024-03-25T17:36:56.731 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.730+0000 7fa2157fa640 10 client.?.objecter ms_handle_connect 0x7fa22006c380 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.730+0000 7fa2157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.730+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409104142 0 0) 0x7fa20c004eb0 con 0x7fa2200b12d0 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2201c0ef0 con 0x7fa2200b12d0 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa218001cb0 con 0x7fa2200b12d0 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 10 monclient(hunting): dump: 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:56.732 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:56.733 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:56.733 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:56.733 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:56.733 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.731+0000 7fa2157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:56.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.738+0000 7fa2157fa640 10 client.?.objecter ms_handle_connect 0x7fa2200aedf0 2024-03-25T17:36:56.739 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.738+0000 7fa2157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3431767768 0 0) 0x7fa2201c0ef0 con 0x7fa2200b12d0 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 >> v1:172.21.15.203:6789/0 conn(0x7fa2200aedf0 legacy=0x7fa2201c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 >> v1:172.21.15.96:6789/0 conn(0x7fa22006c380 legacy=0x7fa2201c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 10 monclient: _finish_hunting 0 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 1 monclient: found mon.c 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2201c2380 con 0x7fa2200b12d0 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 10 monclient: _finish_auth 0 2024-03-25T17:36:56.745 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 10 monclient: _check_auth_tickets 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa2157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 5 monclient: authenticate success, global_id 4568 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 10 monclient: _renew_subs 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2201d7560 con 0x7fa2200b12d0 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 10 monclient: _renew_subs 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2201c1930 con 0x7fa2200b12d0 2024-03-25T17:36:56.746 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.744+0000 7fa226ad9640 1 librados: init done 2024-03-25T17:36:56.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.745+0000 7fa226ad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:56.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.745+0000 7fa226ad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:56.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.745+0000 7fa226ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:56.747 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:56.745+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2201d7750 con 0x7fa2200b12d0 2024-03-25T17:36:57.045 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa218004b60 con 0x7fa2200b12d0 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fa218004f30 con 0x7fa2200b12d0 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa2157fa640 10 monclient: dump: 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:36:57.046 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.044+0000 7fa216ffd640 4 set_mon_vals no callback set 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.045+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fa218005150 con 0x7fa2200b12d0 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.046+0000 7fa224875640 10 monclient: get_auth_request con 0x7fa20c06dd90 auth_method 0 2024-03-25T17:36:57.047 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.046+0000 7fa2157fa640 10 client.4568.objecter ms_handle_connect 0x7fa20c06dd90 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fa218054bb0 con 0x7fa2200b12d0 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 10 client.4568.objecter ms_dispatch 0x7fa220000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2180506c0 con 0x7fa2200b12d0 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:57.054 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.053+0000 7fa2157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:57.300 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.300+0000 7fa226ad9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:57.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.300+0000 7fa226ad9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:57.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.300+0000 7fa226ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:36:57.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.300+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2201c2510 con 0x7fa2200b12d0 2024-03-25T17:36:57.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.327+0000 7fa2157fa640 1 -- 172.21.15.96:0/783704671 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fa2180525f0 con 0x7fa2200b12d0 2024-03-25T17:36:57.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.327+0000 7fa2157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:57.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.327+0000 7fa2157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:57.328 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:36:57.328 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 10 librados: watch_flush enter 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 10 librados: watch_flush exit 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 10 client.4568.objecter successfully canceled tick 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 >> v1:172.21.15.203:6800/3611964061 conn(0x7fa20c06dd90 legacy=0x7fa20c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:57.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 10 monclient: shutdown 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 >> v1:172.21.15.96:6790/0 conn(0x7fa2200b12d0 legacy=0x7fa2201c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 shutdown_connections 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 >> 172.21.15.96:0/783704671 conn(0x7fa2200b5d10 msgr2=0x7fa2201cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 shutdown_connections 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 -- 172.21.15.96:0/783704671 wait complete. 2024-03-25T17:36:57.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:36:57.330+0000 7fa226ad9640 1 librados: shutdown 2024-03-25T17:36:57.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:04.151 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:04.263 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.264+0000 7fc39dd58640 10 monclient: get_monmap_and_config 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.264+0000 7fc39dd58640 10 monclient: build_initial_monmap 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.264+0000 7fc39dd58640 10 monclient: monmap: 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:04.264326+0000 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:04.264326+0000 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.264 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:04.265 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:04.265 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:04.265 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.264+0000 7fc39dd58640 1 Processor -- start 2024-03-25T17:37:04.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 1 -- start start 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient: init 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-b con 0x7fc398061890 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-a con 0x7fc39806e740 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-c con 0x7fc3980ae280 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980ae830 con 0x7fc39806e740 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39806f720 con 0x7fc3980ae280 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980ada80 con 0x7fc398061890 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:04.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.265+0000 7fc39dd58640 10 monclient(hunting): authenticate will time out at 5782.546644s 2024-03-25T17:37:04.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.276+0000 7fc396ffd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc39806e740 0x7fc398145820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59582/0 (socket says 172.21.15.96:59582) 2024-03-25T17:37:04.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.276+0000 7fc396ffd640 1 -- 172.21.15.96:0/774471769 learned_addr learned my addr 172.21.15.96:0/774471769 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514125940 0 0) 0x7fc3980ae830 con 0x7fc39806e740 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 10 monclient(hunting): _negotiate 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T17:37:04.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.283+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380003520 con 0x7fc39806e740 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280194194 0 0) 0x7fc3980ada80 con 0x7fc398061890 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 10 monclient(hunting): _negotiate 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T17:37:04.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.284+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3980ae830 con 0x7fc398061890 2024-03-25T17:37:04.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3739609186 0 0) 0x7fc3980ae830 con 0x7fc398061890 2024-03-25T17:37:04.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 >> v1:172.21.15.203:6789/0 conn(0x7fc3980ae280 legacy=0x7fc398145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:04.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 >> v1:172.21.15.96:6789/0 conn(0x7fc39806e740 legacy=0x7fc398145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:04.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 10 monclient: _finish_hunting 0 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 monclient: found mon.noname-b 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39813fbd0 con 0x7fc398061890 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 10 monclient: _finish_auth 0 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 10 monclient: _check_auth_tickets 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc388001670 con 0x7fc398061890 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.295+0000 7fc39dd58640 5 monclient: authenticate success, global_id 4571 2024-03-25T17:37:04.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.296+0000 7fc3967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.296+0000 7fc3967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.296+0000 7fc3967fc640 10 monclient: dump: 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.298 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.299 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.299 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.299 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.299 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.296+0000 7fc39dd58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:04.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc388001890 con 0x7fc398061890 2024-03-25T17:37:04.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 1 -- 172.21.15.96:0/774471769 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc388001ba0 con 0x7fc398061890 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 4 set_mon_vals no callback set 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc3967fc640 10 monclient: dump: 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.303 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.304 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 10 monclient: shutdown 2024-03-25T17:37:04.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 1 -- 172.21.15.96:0/774471769 >> v1:172.21.15.96:6790/0 conn(0x7fc398061890 legacy=0x7fc39813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 1 -- 172.21.15.96:0/774471769 shutdown_connections 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 1 -- 172.21.15.96:0/774471769 >> 172.21.15.96:0/774471769 conn(0x7fc3980b0ef0 msgr2=0x7fc3980b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.303+0000 7fc39dd58640 1 -- 172.21.15.96:0/774471769 shutdown_connections 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.304+0000 7fc39dd58640 1 -- 172.21.15.96:0/774471769 wait complete. 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.304+0000 7fc39dd58640 10 monclient: build_initial_monmap 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.304+0000 7fc39dd58640 10 monclient: monmap: 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:04.308 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 librados: starting msgr at 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 librados: starting objecter 2024-03-25T17:37:04.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 Processor -- start 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 -- start start 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 librados: setting wanted keys 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 1 librados: calling monclient init 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.305+0000 7fc39dd58640 10 monclient: init 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-a con 0x7fc398061890 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-c con 0x7fc3980ae280 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient(hunting): picked mon.noname-b con 0x7fc3981d4e40 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc398143030 con 0x7fc398061890 2024-03-25T17:37:04.310 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3981d3f70 con 0x7fc3981d4e40 2024-03-25T17:37:04.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3981d4210 con 0x7fc3980ae280 2024-03-25T17:37:04.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:04.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.306+0000 7fc39dd58640 10 monclient(hunting): authenticate will time out at 5782.587543s 2024-03-25T17:37:04.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.310+0000 7fc3977fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc398061890 0x7fc398142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59604/0 (socket says 172.21.15.96:59604) 2024-03-25T17:37:04.311 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.310+0000 7fc3977fe640 1 -- 172.21.15.96:0/2078179717 learned_addr learned my addr 172.21.15.96:0/2078179717 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:04.312 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.310+0000 7fc37ffff640 10 client.?.objecter ms_handle_connect 0x7fc398061890 2024-03-25T17:37:04.312 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.310+0000 7fc37ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:04.317 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.317+0000 7fc37ffff640 10 client.?.objecter ms_handle_connect 0x7fc3980ae280 2024-03-25T17:37:04.317 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.317+0000 7fc37ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060709336 0 0) 0x7fc398143030 con 0x7fc398061890 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 10 monclient(hunting): my global_id is 4587 2024-03-25T17:37:04.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.320+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc384004eb0 con 0x7fc398061890 2024-03-25T17:37:04.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.322+0000 7fc37ffff640 10 client.?.objecter ms_handle_connect 0x7fc3981d4e40 2024-03-25T17:37:04.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.322+0000 7fc37ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214146660 0 0) 0x7fc3981d3f70 con 0x7fc3981d4e40 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 10 monclient(hunting): my global_id is 4570 2024-03-25T17:37:04.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.333+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc398143030 con 0x7fc3981d4e40 2024-03-25T17:37:04.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346792183 0 0) 0x7fc3981d4210 con 0x7fc3980ae280 2024-03-25T17:37:04.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:04.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:04.336 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:04.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 10 monclient(hunting): my global_id is 4574 2024-03-25T17:37:04.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.336+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3981d3f70 con 0x7fc3980ae280 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 939973845 0 0) 0x7fc384004eb0 con 0x7fc398061890 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3981d4210 con 0x7fc398061890 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc388001c60 con 0x7fc398061890 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 10 monclient(hunting): dump: 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.354 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.353+0000 7fc37ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462159349 0 0) 0x7fc3981d3f70 con 0x7fc3980ae280 2024-03-25T17:37:04.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc384004eb0 con 0x7fc3980ae280 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc38c001950 con 0x7fc3980ae280 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.354+0000 7fc37ffff640 10 monclient(hunting): dump: 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.356 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.357 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.357 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.357 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76065314 0 0) 0x7fc398143030 con 0x7fc3981d4e40 2024-03-25T17:37:04.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3981d3f70 con 0x7fc3981d4e40 2024-03-25T17:37:04.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc390001b00 con 0x7fc3981d4e40 2024-03-25T17:37:04.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.362 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.362+0000 7fc37ffff640 10 monclient(hunting): dump: 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.363 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588719864 0 0) 0x7fc384004eb0 con 0x7fc3980ae280 2024-03-25T17:37:04.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 >> v1:172.21.15.203:6789/0 conn(0x7fc3981d4e40 legacy=0x7fc3981e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:04.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 >> v1:172.21.15.96:6789/0 conn(0x7fc398061890 legacy=0x7fc398142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:04.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:04.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 1 monclient: found mon.c 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398145380 con 0x7fc3980ae280 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 10 monclient: _finish_auth 0 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc37ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 5 monclient: authenticate success, global_id 4574 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 10 monclient: _renew_subs 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc398143890 con 0x7fc3980ae280 2024-03-25T17:37:04.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 10 monclient: _renew_subs 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3981e3640 con 0x7fc3980ae280 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.363+0000 7fc39dd58640 1 librados: init done 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.364+0000 7fc39dd58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.364+0000 7fc39dd58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.364+0000 7fc39dd58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.364+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc398144cc0 con 0x7fc3980ae280 2024-03-25T17:37:04.703 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc38c0047e0 con 0x7fc3980ae280 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc38c001d80 con 0x7fc3980ae280 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc37ffff640 10 monclient: dump: 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:04.704 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.703+0000 7fc395ffb640 4 set_mon_vals no callback set 2024-03-25T17:37:04.705 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.704+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fc38c01a9e0 con 0x7fc3980ae280 2024-03-25T17:37:04.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.704+0000 7fc3977fe640 10 monclient: get_auth_request con 0x7fc38406e1a0 auth_method 0 2024-03-25T17:37:04.706 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.704+0000 7fc37ffff640 10 client.4574.objecter ms_handle_connect 0x7fc38406e1a0 2024-03-25T17:37:04.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fc38c054730 con 0x7fc3980ae280 2024-03-25T17:37:04.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 10 client.4574.objecter ms_dispatch 0x7fc398000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:04.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:04.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:04.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-25T17:37:04.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc38c050070 con 0x7fc3980ae280 2024-03-25T17:37:04.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:04.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.709+0000 7fc37ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:04.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.924+0000 7fc39dd58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:04.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.924+0000 7fc39dd58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:04.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.924+0000 7fc39dd58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:04.925 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:04.924+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3980627b0 con 0x7fc3980ae280 2024-03-25T17:37:05.000 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.000+0000 7fc37ffff640 1 -- 172.21.15.96:0/2078179717 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fc38c052170 con 0x7fc3980ae280 2024-03-25T17:37:05.000 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.000+0000 7fc37ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:05.000 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.000+0000 7fc37ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:05.001 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:05.001 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.002+0000 7fc39dd58640 10 librados: watch_flush enter 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 10 librados: watch_flush exit 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 10 client.4574.objecter successfully canceled tick 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 >> v1:172.21.15.203:6800/3611964061 conn(0x7fc38406e1a0 legacy=0x7fc384070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 10 monclient: shutdown 2024-03-25T17:37:05.004 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 >> v1:172.21.15.96:6790/0 conn(0x7fc3980ae280 legacy=0x7fc3981d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 shutdown_connections 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 >> 172.21.15.96:0/2078179717 conn(0x7fc3980b0ef0 msgr2=0x7fc39813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 shutdown_connections 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.003+0000 7fc39dd58640 1 -- 172.21.15.96:0/2078179717 wait complete. 2024-03-25T17:37:05.005 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:05.004+0000 7fc39dd58640 1 librados: shutdown 2024-03-25T17:37:05.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:06.635 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:30.861 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.684 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:30.911 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.713 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:30.940 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.731 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:30.947 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.735 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:30.950 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.747 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:30.974 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.764 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:30.990 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.767 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:30.983 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:06.798 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:31.024 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:11.823 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 10 monclient: get_monmap_and_config 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 10 monclient: build_initial_monmap 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 10 monclient: monmap: 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:11.943454+0000 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:11.943454+0000 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:11.943 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:11.944 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:11.944 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:11.944 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:11.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 1 Processor -- start 2024-03-25T17:37:11.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 1 -- start start 2024-03-25T17:37:11.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.943+0000 7f08694eb640 10 monclient: init 2024-03-25T17:37:11.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:11.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-b con 0x7f08640b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-a con 0x7f08640b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-c con 0x7f086414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640af940 con 0x7f08640b1730 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640b0a70 con 0x7f086414be90 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640b0d50 con 0x7f08640b4d60 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:11.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.944+0000 7f08694eb640 10 monclient(hunting): authenticate will time out at 5790.225443s 2024-03-25T17:37:11.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.954+0000 7f08637fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f086414be90 0x7f086414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:33174/0 (socket says 172.21.15.96:33174) 2024-03-25T17:37:11.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.954+0000 7f08637fe640 1 -- 172.21.15.96:0/3582415973 learned_addr learned my addr 172.21.15.96:0/3582415973 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596397251 0 0) 0x7f08640af940 con 0x7f08640b1730 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 10 monclient(hunting): my global_id is 4590 2024-03-25T17:37:11.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.963+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c003520 con 0x7f08640b1730 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891024963 0 0) 0x7f08640b0a70 con 0x7f086414be90 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 10 monclient(hunting): my global_id is 4573 2024-03-25T17:37:11.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.972+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08640af940 con 0x7f086414be90 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3837594374 0 0) 0x7f08640af940 con 0x7f086414be90 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 >> v1:172.21.15.96:6790/0 conn(0x7f08640b4d60 legacy=0x7f086413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 >> v1:172.21.15.96:6789/0 conn(0x7f08640b1730 legacy=0x7f08640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 monclient: found mon.noname-c 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640b1030 con 0x7f086414be90 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:11.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f085c0019f0 con 0x7f086414be90 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f08694eb640 5 monclient: authenticate success, global_id 4573 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f08694eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f0861ffb640 10 monclient: dump: 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:11.981 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:11.982 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:11.982 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:11.982 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:11.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.979+0000 7f08694eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f085c001b50 con 0x7f086414be90 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 1 -- 172.21.15.96:0/3582415973 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f085c001f20 con 0x7f086414be90 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f0861ffb640 10 monclient: dump: 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:11.987 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f08694eb640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:11.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.987+0000 7f08694eb640 4 set_mon_vals no callback set 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 10 monclient: shutdown 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 1 -- 172.21.15.96:0/3582415973 >> v1:172.21.15.203:6789/0 conn(0x7f086414be90 legacy=0x7f086414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 1 -- 172.21.15.96:0/3582415973 shutdown_connections 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 1 -- 172.21.15.96:0/3582415973 >> 172.21.15.96:0/3582415973 conn(0x7f08640617a0 msgr2=0x7f0864061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 1 -- 172.21.15.96:0/3582415973 shutdown_connections 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.988+0000 7f08694eb640 1 -- 172.21.15.96:0/3582415973 wait complete. 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 10 monclient: build_initial_monmap 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 10 monclient: monmap: 2024-03-25T17:37:11.991 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 librados: starting msgr at 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 librados: starting objecter 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 Processor -- start 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 -- start start 2024-03-25T17:37:11.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 librados: setting wanted keys 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 1 librados: calling monclient init 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.989+0000 7f08694eb640 10 monclient: init 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-c con 0x7f08640b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-a con 0x7f08640b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient(hunting): picked mon.noname-b con 0x7f086414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08641406f0 con 0x7f08640b4d60 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0864140380 con 0x7f086414be90 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0864144ce0 con 0x7f08640b1730 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:11.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.990+0000 7f08694eb640 10 monclient(hunting): authenticate will time out at 5790.271773s 2024-03-25T17:37:11.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.996+0000 7f08627fc640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f08640b4d60 0x7f08641d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41172/0 (socket says 172.21.15.96:41172) 2024-03-25T17:37:11.996 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.996+0000 7f08627fc640 1 -- 172.21.15.96:0/437313387 learned_addr learned my addr 172.21.15.96:0/437313387 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:11.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.997+0000 7f084b7fe640 10 client.?.objecter ms_handle_connect 0x7f08640b4d60 2024-03-25T17:37:11.997 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:11.997+0000 7f084b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:12.002 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.003+0000 7f084b7fe640 10 client.?.objecter ms_handle_connect 0x7f086414be90 2024-03-25T17:37:12.002 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.003+0000 7f084b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229876049 0 0) 0x7f08641406f0 con 0x7f08640b4d60 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0850004eb0 con 0x7f08640b4d60 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 client.?.objecter ms_handle_connect 0x7f08640b1730 2024-03-25T17:37:12.006 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.006+0000 7f084b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.008+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444502888 0 0) 0x7f0864140380 con 0x7f086414be90 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.009+0000 7f084b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.009+0000 7f084b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.009+0000 7f084b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.009+0000 7f084b7fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T17:37:12.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.009+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08641406f0 con 0x7f086414be90 2024-03-25T17:37:12.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041799386 0 0) 0x7f0850004eb0 con 0x7f08640b4d60 2024-03-25T17:37:12.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0864140380 con 0x7f08640b4d60 2024-03-25T17:37:12.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f0858001990 con 0x7f08640b4d60 2024-03-25T17:37:12.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 10 monclient(hunting): dump: 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:12.014 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:12.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.013+0000 7f084b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:12.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315517721 0 0) 0x7f0864144ce0 con 0x7f08640b1730 2024-03-25T17:37:12.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:12.026 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:12.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:12.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T17:37:12.027 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.026+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0850004eb0 con 0x7f08640b1730 2024-03-25T17:37:12.031 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2363725650 0 0) 0x7f0864140380 con 0x7f08640b4d60 2024-03-25T17:37:12.031 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 >> v1:172.21.15.96:6790/0 conn(0x7f08640b1730 legacy=0x7f0864131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 >> v1:172.21.15.203:6789/0 conn(0x7f086414be90 legacy=0x7f08641d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 1 monclient: found mon.a 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0864132520 con 0x7f08640b4d60 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f084b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 5 monclient: authenticate success, global_id 4593 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 10 monclient: _renew_subs 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864132770 con 0x7f08640b4d60 2024-03-25T17:37:12.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 10 monclient: _renew_subs 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f086413e130 con 0x7f08640b4d60 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.030+0000 7f08694eb640 1 librados: init done 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.031+0000 7f08694eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.031+0000 7f08694eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.031+0000 7f08694eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:12.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.031+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864140880 con 0x7f08640b4d60 2024-03-25T17:37:12.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0858003610 con 0x7f08640b4d60 2024-03-25T17:37:12.339 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f08580039e0 con 0x7f08640b4d60 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f084b7fe640 10 monclient: dump: 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:12.340 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.339+0000 7f08617fa640 4 set_mon_vals no callback set 2024-03-25T17:37:12.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.340+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f0858003c00 con 0x7f08640b4d60 2024-03-25T17:37:12.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.341+0000 7f0862ffd640 10 monclient: get_auth_request con 0x7f085006de00 auth_method 0 2024-03-25T17:37:12.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.341+0000 7f084b7fe640 10 client.4593.objecter ms_handle_connect 0x7f085006de00 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f08580547b0 con 0x7f08640b4d60 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 10 client.4593.objecter ms_dispatch 0x7f0864000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08580502c0 con 0x7f08640b4d60 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:12.346 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.346+0000 7f084b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:12.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.589+0000 7f08694eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:12.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.589+0000 7f08694eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:12.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.589+0000 7f08694eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:12.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.589+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0864140510 con 0x7f08640b4d60 2024-03-25T17:37:12.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.620+0000 7f084b7fe640 1 -- 172.21.15.96:0/437313387 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f08580521f0 con 0x7f08640b4d60 2024-03-25T17:37:12.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.620+0000 7f084b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:12.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.620+0000 7f084b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:12.620 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:12.621 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:12.623 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 10 librados: watch_flush enter 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 10 librados: watch_flush exit 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 10 client.4593.objecter successfully canceled tick 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 >> v1:172.21.15.203:6800/3611964061 conn(0x7f085006de00 legacy=0x7f08500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 10 monclient: shutdown 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 >> v1:172.21.15.96:6789/0 conn(0x7f08640b4d60 legacy=0x7f08641d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 shutdown_connections 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 >> 172.21.15.96:0/437313387 conn(0x7f08640617a0 msgr2=0x7f086413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:12.624 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 shutdown_connections 2024-03-25T17:37:12.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 -- 172.21.15.96:0/437313387 wait complete. 2024-03-25T17:37:12.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:12.623+0000 7f08694eb640 1 librados: shutdown 2024-03-25T17:37:12.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:19.443 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:19.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.556+0000 7f5bd980d640 10 monclient: get_monmap_and_config 2024-03-25T17:37:19.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.556+0000 7f5bd980d640 10 monclient: build_initial_monmap 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.556+0000 7f5bd980d640 10 monclient: monmap: 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:19.558158+0000 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:19.558158+0000 2024-03-25T17:37:19.558 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:19.559 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:19.559 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:19.559 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:19.559 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:19.559 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.557+0000 7f5bd980d640 1 Processor -- start 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.557+0000 7f5bd980d640 1 -- start start 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.557+0000 7f5bd980d640 10 monclient: init 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:19.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4142520 con 0x7f5bd413f710 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b28a0 con 0x7f5bd41430c0 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b2a30 con 0x7f5bd40a98f0 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:19.563 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.558+0000 7f5bd980d640 10 monclient(hunting): authenticate will time out at 5797.840218s 2024-03-25T17:37:19.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.572+0000 7f5bd37fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5bd41430c0 0x7f5bd414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:43866/0 (socket says 172.21.15.96:43866) 2024-03-25T17:37:19.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.572+0000 7f5bd37fe640 1 -- 172.21.15.96:0/4240931361 learned_addr learned my addr 172.21.15.96:0/4240931361 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:19.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611081941 0 0) 0x7f5bd40b28a0 con 0x7f5bd41430c0 2024-03-25T17:37:19.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:19.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:19.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:19.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 10 monclient(hunting): my global_id is 4579 2024-03-25T17:37:19.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.583+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc003520 con 0x7f5bd41430c0 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.588+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393141173 0 0) 0x7f5bd4142520 con 0x7f5bd413f710 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.589+0000 7f5bd1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.589+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.589+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.589+0000 7f5bd1ffb640 10 monclient(hunting): my global_id is 4596 2024-03-25T17:37:19.590 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.589+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40b28a0 con 0x7f5bd413f710 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255629655 0 0) 0x7f5bd40b2a30 con 0x7f5bd40a98f0 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T17:37:19.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.592+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd4142520 con 0x7f5bd40a98f0 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3175429763 0 0) 0x7f5bd40b28a0 con 0x7f5bd413f710 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 >> v1:172.21.15.96:6790/0 conn(0x7f5bd40a98f0 legacy=0x7f5bd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 >> v1:172.21.15.203:6789/0 conn(0x7f5bd41430c0 legacy=0x7f5bd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 monclient: found mon.noname-a 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:19.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40b44c0 con 0x7f5bd413f710 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd980d640 5 monclient: authenticate success, global_id 4596 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd980d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5bc8001630 con 0x7f5bd413f710 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:19.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd1ffb640 10 monclient: dump: 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:19.597 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.593+0000 7f5bd980d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:19.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc8001880 con 0x7f5bd413f710 2024-03-25T17:37:19.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:19.612 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 1 -- 172.21.15.96:0/4240931361 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5bc8001c10 con 0x7f5bd413f710 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd980d640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd980d640 4 set_mon_vals no callback set 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.611+0000 7f5bd1ffb640 10 monclient: dump: 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.613 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:19.614 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:19.614 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:19.614 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:19.614 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:19.614 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 10 monclient: shutdown 2024-03-25T17:37:19.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:19.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 1 -- 172.21.15.96:0/4240931361 >> v1:172.21.15.96:6789/0 conn(0x7f5bd413f710 legacy=0x7f5bd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:19.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 1 -- 172.21.15.96:0/4240931361 shutdown_connections 2024-03-25T17:37:19.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 1 -- 172.21.15.96:0/4240931361 >> 172.21.15.96:0/4240931361 conn(0x7f5bd40b7ea0 msgr2=0x7f5bd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 1 -- 172.21.15.96:0/4240931361 shutdown_connections 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.612+0000 7f5bd980d640 1 -- 172.21.15.96:0/4240931361 wait complete. 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 10 monclient: build_initial_monmap 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 10 monclient: monmap: 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:19.618 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 librados: starting msgr at 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 librados: starting objecter 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 Processor -- start 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 -- start start 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 librados: setting wanted keys 2024-03-25T17:37:19.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 1 librados: calling monclient init 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.613+0000 7f5bd980d640 10 monclient: init 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd414e5c0 con 0x7f5bd41430c0 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd413ad30 con 0x7f5bd40a98f0 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4141c70 con 0x7f5bd413f710 2024-03-25T17:37:19.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:19.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.614+0000 7f5bd980d640 10 monclient(hunting): authenticate will time out at 5797.896655s 2024-03-25T17:37:19.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.626+0000 7f5bd27fc640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f5bd413f710 0x7f5bd4141560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:50934/0 (socket says 172.21.15.96:50934) 2024-03-25T17:37:19.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.626+0000 7f5bd27fc640 1 -- 172.21.15.96:0/362086801 learned_addr learned my addr 172.21.15.96:0/362086801 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:19.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.628+0000 7f5bbbfff640 10 client.?.objecter ms_handle_connect 0x7f5bd413f710 2024-03-25T17:37:19.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.628+0000 7f5bbbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:19.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290432713 0 0) 0x7f5bd4141c70 con 0x7f5bd413f710 2024-03-25T17:37:19.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 monclient(hunting): my global_id is 4586 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0004eb0 con 0x7f5bd413f710 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 client.?.objecter ms_handle_connect 0x7f5bd40a98f0 2024-03-25T17:37:19.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.636+0000 7f5bbbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:19.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.640+0000 7f5bbbfff640 10 client.?.objecter ms_handle_connect 0x7f5bd41430c0 2024-03-25T17:37:19.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.640+0000 7f5bbbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:19.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844845226 0 0) 0x7f5bc0004eb0 con 0x7f5bd413f710 2024-03-25T17:37:19.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd4141c70 con 0x7f5bd413f710 2024-03-25T17:37:19.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5bc8001c50 con 0x7f5bd413f710 2024-03-25T17:37:19.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 10 monclient(hunting): dump: 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:19.645 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:19.646 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:19.646 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.643+0000 7f5bbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:19.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026548688 0 0) 0x7f5bd413ad30 con 0x7f5bd40a98f0 2024-03-25T17:37:19.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:19.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:19.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:19.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 10 monclient(hunting): my global_id is 4582 2024-03-25T17:37:19.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.650+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0004eb0 con 0x7f5bd40a98f0 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2046366070 0 0) 0x7f5bd4141c70 con 0x7f5bd413f710 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 >> v1:172.21.15.203:6789/0 conn(0x7f5bd40a98f0 legacy=0x7f5bd414d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 >> v1:172.21.15.96:6789/0 conn(0x7f5bd41430c0 legacy=0x7f5bd414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 1 monclient: found mon.c 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:19.658 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd414e360 con 0x7f5bd413f710 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 10 monclient: _finish_auth 0 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 5 monclient: authenticate success, global_id 4586 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 10 monclient: _renew_subs 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40b6740 con 0x7f5bd413f710 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:19.659 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 10 monclient: _renew_subs 2024-03-25T17:37:19.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:19.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd4139100 con 0x7f5bd413f710 2024-03-25T17:37:19.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.656+0000 7f5bd980d640 1 librados: init done 2024-03-25T17:37:19.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.657+0000 7f5bd980d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:19.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.657+0000 7f5bd980d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:19.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.657+0000 7f5bd980d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:19.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.657+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd40b3ef0 con 0x7f5bd413f710 2024-03-25T17:37:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc80038a0 con 0x7f5bd413f710 2024-03-25T17:37:19.958 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5bc8003cb0 con 0x7f5bd413f710 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bbbfff640 10 monclient: dump: 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:19.959 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:19.960 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:19.960 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:19.960 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:19.960 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:19.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.957+0000 7f5bd17fa640 4 set_mon_vals no callback set 2024-03-25T17:37:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.958+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5bc8003ed0 con 0x7f5bd413f710 2024-03-25T17:37:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.958+0000 7f5bd2ffd640 10 monclient: get_auth_request con 0x7f5bc006ddd0 auth_method 0 2024-03-25T17:37:19.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.959+0000 7f5bbbfff640 10 client.4586.objecter ms_handle_connect 0x7f5bc006ddd0 2024-03-25T17:37:19.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5bc8054980 con 0x7f5bd413f710 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 10 client.4586.objecter ms_dispatch 0x7f5bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bc8050490 con 0x7f5bd413f710 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:19.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:19.964+0000 7f5bbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:20.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.214+0000 7f5bd980d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:20.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.214+0000 7f5bd980d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:20.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.214+0000 7f5bd980d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:20.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.214+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bd413a960 con 0x7f5bd413f710 2024-03-25T17:37:20.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.242+0000 7f5bbbfff640 1 -- 172.21.15.96:0/362086801 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5bc80523c0 con 0x7f5bd413f710 2024-03-25T17:37:20.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.242+0000 7f5bbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:20.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.242+0000 7f5bbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:20.243 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:20.243 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 10 librados: watch_flush enter 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 10 librados: watch_flush exit 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 10 client.4586.objecter successfully canceled tick 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 >> v1:172.21.15.203:6800/3611964061 conn(0x7f5bc006ddd0 legacy=0x7f5bc0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 10 monclient: shutdown 2024-03-25T17:37:20.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 >> v1:172.21.15.96:6790/0 conn(0x7f5bd413f710 legacy=0x7f5bd4141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 shutdown_connections 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 >> 172.21.15.96:0/362086801 conn(0x7f5bd40b7ea0 msgr2=0x7f5bd413d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 shutdown_connections 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.244+0000 7f5bd980d640 1 -- 172.21.15.96:0/362086801 wait complete. 2024-03-25T17:37:20.247 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:20.245+0000 7f5bd980d640 1 librados: shutdown 2024-03-25T17:37:20.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:26.634 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:50.860 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.640 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:50.866 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.691 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:50.906 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.693 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:08:50.920 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.694 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:50.909 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.696 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:50.922 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.711 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:50.937 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.714 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:50.929 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:26.737 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:08:50.963 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:27.065 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:27.178 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.178+0000 7f94f0bed640 10 monclient: get_monmap_and_config 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.178+0000 7f94f0bed640 10 monclient: build_initial_monmap 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.178+0000 7f94f0bed640 10 monclient: monmap: 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:27.179503+0000 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:27.179503+0000 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:27.179 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.180 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.178+0000 7f94f0bed640 1 Processor -- start 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 1 -- start start 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient: init 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-a con 0x7f94ec0b0070 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-c con 0x7f94ec13c170 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-b con 0x7f94ec06fa90 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b0450 con 0x7f94ec0b0070 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec071440 con 0x7f94ec13c170 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b0bf0 con 0x7f94ec06fa90 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:27.181 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.179+0000 7f94f0bed640 10 monclient(hunting): authenticate will time out at 5805.461830s 2024-03-25T17:37:27.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.186+0000 7f94ead9d640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f94ec06fa90 0x7f94ec14a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:59208/0 (socket says 172.21.15.96:59208) 2024-03-25T17:37:27.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.186+0000 7f94ead9d640 1 -- 172.21.15.96:0/2270091882 learned_addr learned my addr 172.21.15.96:0/2270091882 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441888617 0 0) 0x7f94ec0b0450 con 0x7f94ec0b0070 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 10 monclient(hunting): _negotiate 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 10 monclient(hunting): my global_id is 4602 2024-03-25T17:37:27.193 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.193+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d4003520 con 0x7f94ec0b0070 2024-03-25T17:37:27.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140142692 0 0) 0x7f94ec0b0bf0 con 0x7f94ec06fa90 2024-03-25T17:37:27.194 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 10 monclient(hunting): _negotiate 2024-03-25T17:37:27.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:27.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:27.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 10 monclient(hunting): my global_id is 4589 2024-03-25T17:37:27.195 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.194+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec0b0450 con 0x7f94ec06fa90 2024-03-25T17:37:27.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787358652 0 0) 0x7f94d4003520 con 0x7f94ec0b0070 2024-03-25T17:37:27.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 >> v1:172.21.15.96:6790/0 conn(0x7f94ec06fa90 legacy=0x7f94ec14a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 >> v1:172.21.15.203:6789/0 conn(0x7f94ec13c170 legacy=0x7f94ec141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: _finish_hunting 0 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 monclient: found mon.noname-a 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0716b0 con 0x7f94ec0b0070 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: _finish_auth 0 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: _check_auth_tickets 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94dc001690 con 0x7f94ec0b0070 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94f0bed640 5 monclient: authenticate success, global_id 4602 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94f0bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94e959a640 10 monclient: dump: 2024-03-25T17:37:27.217 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.215+0000 7f94f0bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:27.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.226+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94dc0017f0 con 0x7f94ec0b0070 2024-03-25T17:37:27.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.226+0000 7f94e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:27.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.226+0000 7f94e959a640 1 -- 172.21.15.96:0/2270091882 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94dc001c00 con 0x7f94ec0b0070 2024-03-25T17:37:27.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.226+0000 7f94e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:27.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94e959a640 10 monclient: dump: 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94f0bed640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:27.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94f0bed640 4 set_mon_vals no callback set 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94f0bed640 10 monclient: shutdown 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94f0bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.227+0000 7f94f0bed640 1 -- 172.21.15.96:0/2270091882 >> v1:172.21.15.96:6789/0 conn(0x7f94ec0b0070 legacy=0x7f94ec13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 1 -- 172.21.15.96:0/2270091882 shutdown_connections 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 1 -- 172.21.15.96:0/2270091882 >> 172.21.15.96:0/2270091882 conn(0x7f94ec0b0ef0 msgr2=0x7f94ec0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 1 -- 172.21.15.96:0/2270091882 shutdown_connections 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 1 -- 172.21.15.96:0/2270091882 wait complete. 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 10 monclient: build_initial_monmap 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.228+0000 7f94f0bed640 10 monclient: monmap: 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:27.230 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 librados: starting msgr at 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 librados: starting objecter 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 Processor -- start 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 -- start start 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 librados: setting wanted keys 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 1 librados: calling monclient init 2024-03-25T17:37:27.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.229+0000 7f94f0bed640 10 monclient: init 2024-03-25T17:37:27.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-a con 0x7f94ec06fa90 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-b con 0x7f94ec0b0070 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient(hunting): picked mon.noname-c con 0x7f94ec13c170 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec141c20 con 0x7f94ec06fa90 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec140300 con 0x7f94ec0b0070 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec140de0 con 0x7f94ec13c170 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:27.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.230+0000 7f94f0bed640 10 monclient(hunting): authenticate will time out at 5805.512213s 2024-03-25T17:37:27.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.243+0000 7f94e9d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f94ec0b0070 0x7f94ec1daaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:34098/0 (socket says 172.21.15.96:34098) 2024-03-25T17:37:27.244 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.243+0000 7f94e9d9b640 1 -- 172.21.15.96:0/810261512 learned_addr learned my addr 172.21.15.96:0/810261512 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:27.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.247+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec13c170 2024-03-25T17:37:27.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.247+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:27.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.248+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec0b0070 2024-03-25T17:37:27.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.248+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983339589 0 0) 0x7f94ec140300 con 0x7f94ec0b0070 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 10 monclient(hunting): my global_id is 4588 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.257+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc004eb0 con 0x7f94ec0b0070 2024-03-25T17:37:27.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905513344 0 0) 0x7f94ec140de0 con 0x7f94ec13c170 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 monclient(hunting): my global_id is 4592 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec140300 con 0x7f94ec13c170 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 client.?.objecter ms_handle_connect 0x7f94ec06fa90 2024-03-25T17:37:27.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.258+0000 7f94daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076989260 0 0) 0x7f94cc004eb0 con 0x7f94ec0b0070 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec140de0 con 0x7f94ec0b0070 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e0001960 con 0x7f94ec0b0070 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:27.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 10 monclient(hunting): dump: 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.280+0000 7f94daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683387482 0 0) 0x7f94ec141c20 con 0x7f94ec06fa90 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 10 monclient(hunting): my global_id is 4605 2024-03-25T17:37:27.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.285+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc004eb0 con 0x7f94ec06fa90 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 790423177 0 0) 0x7f94ec140300 con 0x7f94ec13c170 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec141c20 con 0x7f94ec13c170 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e4001cb0 con 0x7f94ec13c170 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.286+0000 7f94daffd640 10 monclient(hunting): dump: 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.287 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:27.288 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:27.288 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:27.288 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3144247572 0 0) 0x7f94ec140de0 con 0x7f94ec0b0070 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 >> v1:172.21.15.96:6790/0 conn(0x7f94ec13c170 legacy=0x7f94ec1e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 >> v1:172.21.15.96:6789/0 conn(0x7f94ec06fa90 legacy=0x7f94ec1364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 10 monclient: _finish_hunting 0 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 1 monclient: found mon.b 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.291+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec140f70 con 0x7f94ec0b0070 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94daffd640 10 monclient: _finish_auth 0 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94daffd640 10 monclient: _check_auth_tickets 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 5 monclient: authenticate success, global_id 4588 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 10 monclient: _renew_subs 2024-03-25T17:37:27.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec0adaf0 con 0x7f94ec0b0070 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 10 monclient: _renew_subs 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec136bd0 con 0x7f94ec0b0070 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.292+0000 7f94f0bed640 1 librados: init done 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.293+0000 7f94f0bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:27.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.293+0000 7f94f0bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:27.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.293+0000 7f94f0bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:37:27.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.293+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec140490 con 0x7f94ec0b0070 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e0003610 con 0x7f94ec0b0070 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f94e0003920 con 0x7f94ec0b0070 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 10 monclient: dump: 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:27.621 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:27.622 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94e8d99640 4 set_mon_vals no callback set 2024-03-25T17:37:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.620+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f94e0003b40 con 0x7f94ec0b0070 2024-03-25T17:37:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.621+0000 7f94ea59c640 10 monclient: get_auth_request con 0x7f94cc06df60 auth_method 0 2024-03-25T17:37:27.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.621+0000 7f94daffd640 10 client.4588.objecter ms_handle_connect 0x7f94cc06df60 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f94e0054800 con 0x7f94ec0b0070 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 10 client.4588.objecter ms_dispatch 0x7f94ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94e0050310 con 0x7f94ec0b0070 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:27.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.626+0000 7f94daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:27.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.886+0000 7f94f0bed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:27.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.886+0000 7f94f0bed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:27.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.886+0000 7f94f0bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:37:27.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.886+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94ec0627b0 con 0x7f94ec0b0070 2024-03-25T17:37:27.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.925+0000 7f94daffd640 1 -- 172.21.15.96:0/810261512 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f94e0052240 con 0x7f94ec0b0070 2024-03-25T17:37:27.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.925+0000 7f94daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:27.926 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.925+0000 7f94daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:27.926 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:27.926 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 10 librados: watch_flush enter 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 10 librados: watch_flush exit 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 10 client.4588.objecter successfully canceled tick 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 >> v1:172.21.15.203:6800/3611964061 conn(0x7f94cc06df60 legacy=0x7f94cc070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 10 monclient: shutdown 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.927+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 >> v1:172.21.15.203:6789/0 conn(0x7f94ec0b0070 legacy=0x7f94ec1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.928+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 shutdown_connections 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.928+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 >> 172.21.15.96:0/810261512 conn(0x7f94ec0b0ef0 msgr2=0x7f94ec072260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.928+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 shutdown_connections 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.928+0000 7f94f0bed640 1 -- 172.21.15.96:0/810261512 wait complete. 2024-03-25T17:37:27.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:27.928+0000 7f94f0bed640 1 librados: shutdown 2024-03-25T17:37:27.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:34.749 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.858+0000 7f5cd2e1e640 10 monclient: get_monmap_and_config 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.858+0000 7f5cd2e1e640 10 monclient: build_initial_monmap 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.858+0000 7f5cd2e1e640 10 monclient: monmap: 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:34.859904+0000 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:34.859904+0000 2024-03-25T17:37:34.859 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:34.860 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.860 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:34.860 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:34.860 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:34.860 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 1 Processor -- start 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 1 -- start start 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient: init 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5ccc0b1730 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0af940 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.861 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0b0a70 con 0x7f5ccc14be90 2024-03-25T17:37:34.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0b0d50 con 0x7f5ccc0b1730 2024-03-25T17:37:34.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:34.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.859+0000 7f5cd2e1e640 10 monclient(hunting): authenticate will time out at 5813.141897s 2024-03-25T17:37:34.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.860+0000 7f5cd13bb640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5ccc14be90 0x7f5ccc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:37624/0 (socket says 172.21.15.96:37624) 2024-03-25T17:37:34.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.860+0000 7f5cd13bb640 1 -- 172.21.15.96:0/1510526917 learned_addr learned my addr 172.21.15.96:0/1510526917 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535841006 0 0) 0x7f5ccc0af940 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T17:37:34.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.909+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cbc003520 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718450213 0 0) 0x7f5cbc003520 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 >> v1:172.21.15.96:6790/0 conn(0x7f5ccc0b1730 legacy=0x7f5ccc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 >> v1:172.21.15.203:6789/0 conn(0x7f5ccc14be90 legacy=0x7f5ccc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 monclient: found mon.noname-a 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0b1030 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:34.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cc0001690 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5ccb7fe640 10 monclient: dump: 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5cd2e1e640 5 monclient: authenticate success, global_id 4608 2024-03-25T17:37:34.929 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.925+0000 7f5cd2e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:34.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cc00017f0 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 1 -- 172.21.15.96:0/1510526917 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cc0001c40 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 4 set_mon_vals no callback set 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5ccb7fe640 10 monclient: dump: 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:34.932 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:34.933 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:34.933 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 10 monclient: shutdown 2024-03-25T17:37:34.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:34.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/1510526917 >> v1:172.21.15.96:6789/0 conn(0x7f5ccc0b4d60 legacy=0x7f5ccc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:34.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/1510526917 shutdown_connections 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/1510526917 >> 172.21.15.96:0/1510526917 conn(0x7f5ccc0617a0 msgr2=0x7f5ccc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/1510526917 shutdown_connections 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.931+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/1510526917 wait complete. 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.932+0000 7f5cd2e1e640 10 monclient: build_initial_monmap 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.932+0000 7f5cd2e1e640 10 monclient: monmap: 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:34.935 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.932+0000 7f5cd2e1e640 1 librados: starting msgr at 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.932+0000 7f5cd2e1e640 1 librados: starting objecter 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 Processor -- start 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 -- start start 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 librados: setting wanted keys 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 librados: calling monclient init 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient: init 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc0b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5ccc14be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc13bde0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc140f50 con 0x7f5ccc14be90 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc1410e0 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:34.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.933+0000 7f5cd2e1e640 10 monclient(hunting): authenticate will time out at 5813.215854s 2024-03-25T17:37:34.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.944+0000 7f5cd0bba640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5ccc0b1730 0x7f5ccc140840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40282/0 (socket says 172.21.15.96:40282) 2024-03-25T17:37:34.945 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.944+0000 7f5cd0bba640 1 -- 172.21.15.96:0/2728213231 learned_addr learned my addr 172.21.15.96:0/2728213231 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:34.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.946+0000 7f5cc97fa640 10 client.?.objecter ms_handle_connect 0x7f5ccc0b1730 2024-03-25T17:37:34.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.946+0000 7f5cc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:34.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.947+0000 7f5cc97fa640 10 client.?.objecter ms_handle_connect 0x7f5ccc14be90 2024-03-25T17:37:34.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.947+0000 7f5cc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:34.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986119034 0 0) 0x7f5ccc13bde0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.953 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:34.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:34.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:34.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 10 monclient(hunting): my global_id is 4611 2024-03-25T17:37:34.954 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.953+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb0004eb0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.956+0000 7f5cc97fa640 10 client.?.objecter ms_handle_connect 0x7f5ccc0b4d60 2024-03-25T17:37:34.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.956+0000 7f5cc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220276736 0 0) 0x7f5ccc140f50 con 0x7f5ccc14be90 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T17:37:34.963 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.962+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc13bde0 con 0x7f5ccc14be90 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437935248 0 0) 0x7f5ccc1410e0 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 10 monclient(hunting): my global_id is 4595 2024-03-25T17:37:34.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.963+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc140f50 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995049921 0 0) 0x7f5cb0004eb0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ccc1410e0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cc0001690 con 0x7f5ccc0b1730 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 10 monclient(hunting): dump: 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.973 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:34.974 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:34.974 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:34.974 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.972+0000 7f5cc97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 830648090 0 0) 0x7f5ccc13bde0 con 0x7f5ccc14be90 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cb0004eb0 con 0x7f5ccc14be90 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cc4001b00 con 0x7f5ccc14be90 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.979+0000 7f5cc97fa640 10 monclient(hunting): dump: 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.980 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844912536 0 0) 0x7f5ccc140f50 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ccc13bde0 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cb8001860 con 0x7f5ccc0b4d60 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.981+0000 7f5cc97fa640 10 monclient(hunting): dump: 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:34.982 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:34.983 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:34.986 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4255945420 0 0) 0x7f5ccc1410e0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 >> v1:172.21.15.96:6790/0 conn(0x7f5ccc0b4d60 legacy=0x7f5ccc1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 >> v1:172.21.15.203:6789/0 conn(0x7f5ccc14be90 legacy=0x7f5ccc142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 1 monclient: found mon.a 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc1d8450 con 0x7f5ccc0b1730 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 10 monclient: _finish_auth 0 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 5 monclient: authenticate success, global_id 4611 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 10 monclient: _renew_subs 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:34.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc1d86a0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 10 monclient: _renew_subs 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc1426d0 con 0x7f5ccc0b1730 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.985+0000 7f5cd2e1e640 1 librados: init done 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.986+0000 7f5cd2e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.987+0000 7f5cd2e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:34.988 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.987+0000 7f5cd2e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:34.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:34.987+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc1401f0 con 0x7f5ccc0b1730 2024-03-25T17:37:35.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cc0003960 con 0x7f5ccc0b1730 2024-03-25T17:37:35.293 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5cc0003d30 con 0x7f5ccc0b1730 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5cc97fa640 10 monclient: dump: 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:35.294 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:35.295 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:35.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.292+0000 7f5ccaffd640 4 set_mon_vals no callback set 2024-03-25T17:37:35.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.293+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5cc0003f50 con 0x7f5ccc0b1730 2024-03-25T17:37:35.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.294+0000 7f5ccbfff640 10 monclient: get_auth_request con 0x7f5cb006e0f0 auth_method 0 2024-03-25T17:37:35.296 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.294+0000 7f5cc97fa640 10 client.4611.objecter ms_handle_connect 0x7f5cb006e0f0 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5cc0054970 con 0x7f5ccc0b1730 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 10 client.4611.objecter ms_dispatch 0x7f5ccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T17:37:35.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cc0050480 con 0x7f5ccc0b1730 2024-03-25T17:37:35.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:35.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.300+0000 7f5cc97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:35.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.546+0000 7f5cd2e1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:35.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.546+0000 7f5cd2e1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:35.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.546+0000 7f5cd2e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:35.547 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.546+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ccc0627b0 con 0x7f5ccc0b1730 2024-03-25T17:37:35.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.572+0000 7f5cc97fa640 1 -- 172.21.15.96:0/2728213231 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5cc00523b0 con 0x7f5ccc0b1730 2024-03-25T17:37:35.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.572+0000 7f5cc97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:35.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.572+0000 7f5cc97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:35.574 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:35.574 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 10 librados: watch_flush enter 2024-03-25T17:37:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 10 librados: watch_flush exit 2024-03-25T17:37:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T17:37:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 10 client.4611.objecter successfully canceled tick 2024-03-25T17:37:35.576 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 >> v1:172.21.15.203:6800/3611964061 conn(0x7f5cb006e0f0 legacy=0x7f5cb00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 10 monclient: shutdown 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 >> v1:172.21.15.96:6789/0 conn(0x7f5ccc0b1730 legacy=0x7f5ccc140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 shutdown_connections 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 >> 172.21.15.96:0/2728213231 conn(0x7f5ccc0617a0 msgr2=0x7f5ccc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 shutdown_connections 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 -- 172.21.15.96:0/2728213231 wait complete. 2024-03-25T17:37:35.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:35.575+0000 7f5cd2e1e640 1 librados: shutdown 2024-03-25T17:37:35.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:42.395 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:42.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.510+0000 7fdb01fc9640 10 monclient: get_monmap_and_config 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.510+0000 7fdb01fc9640 10 monclient: build_initial_monmap 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.510+0000 7fdb01fc9640 10 monclient: monmap: 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:42.511635+0000 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:42.511635+0000 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:42.511 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.511+0000 7fdb01fc9640 1 Processor -- start 2024-03-25T17:37:42.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.511+0000 7fdb01fc9640 1 -- start start 2024-03-25T17:37:42.512 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.511+0000 7fdb01fc9640 10 monclient: init 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-a con 0x7fdafc0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fdafc13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fdafc1430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc142520 con 0x7fdafc0a98f0 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0b28a0 con 0x7fdafc13f710 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0b2a30 con 0x7fdafc1430c0 2024-03-25T17:37:42.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:42.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.512+0000 7fdb01fc9640 10 monclient(hunting): authenticate will time out at 5820.794327s 2024-03-25T17:37:42.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.515+0000 7fdafb7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdafc0a98f0 0x7fdafc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45140/0 (socket says 172.21.15.96:45140) 2024-03-25T17:37:42.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.515+0000 7fdafb7fe640 1 -- 172.21.15.96:0/3235123137 learned_addr learned my addr 172.21.15.96:0/3235123137 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710708585 0 0) 0x7fdafc142520 con 0x7fdafc0a98f0 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T17:37:42.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.523+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8003520 con 0x7fdafc0a98f0 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207256816 0 0) 0x7fdae8003520 con 0x7fdafc0a98f0 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 >> v1:172.21.15.96:6790/0 conn(0x7fdafc1430c0 legacy=0x7fdafc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 >> v1:172.21.15.203:6789/0 conn(0x7fdafc13f710 legacy=0x7fdafc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 monclient: found mon.noname-a 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:37:42.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0b44c0 con 0x7fdafc0a98f0 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: _finish_auth 0 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fdaec001690 con 0x7fdafc0a98f0 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdafaffd640 10 monclient: dump: 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:42.536 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:42.537 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:42.537 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdb01fc9640 5 monclient: authenticate success, global_id 4614 2024-03-25T17:37:42.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.534+0000 7fdb01fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:42.545 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaec0017f0 con 0x7fdafc0a98f0 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 1 -- 172.21.15.96:0/3235123137 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fdaec001c00 con 0x7fdafc0a98f0 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdb01fc9640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdb01fc9640 4 set_mon_vals no callback set 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.545+0000 7fdafaffd640 10 monclient: dump: 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:42.546 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:42.547 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:42.547 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:42.547 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 10 monclient: shutdown 2024-03-25T17:37:42.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:42.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 1 -- 172.21.15.96:0/3235123137 >> v1:172.21.15.96:6789/0 conn(0x7fdafc0a98f0 legacy=0x7fdafc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:42.549 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 1 -- 172.21.15.96:0/3235123137 shutdown_connections 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 1 -- 172.21.15.96:0/3235123137 >> 172.21.15.96:0/3235123137 conn(0x7fdafc0b7ea0 msgr2=0x7fdafc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 1 -- 172.21.15.96:0/3235123137 shutdown_connections 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 1 -- 172.21.15.96:0/3235123137 wait complete. 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 10 monclient: build_initial_monmap 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.546+0000 7fdb01fc9640 10 monclient: monmap: 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:42.550 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 librados: starting msgr at 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 librados: starting objecter 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 Processor -- start 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 -- start start 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 librados: setting wanted keys 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 1 librados: calling monclient init 2024-03-25T17:37:42.551 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 10 monclient: init 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.547+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fdafc0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-a con 0x7fdafc13f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fdafc1430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0abd30 con 0x7fdafc13f710 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc1d77a0 con 0x7fdafc0a98f0 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc1d51f0 con 0x7fdafc1430c0 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:42.552 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.548+0000 7fdb01fc9640 10 monclient(hunting): authenticate will time out at 5820.830115s 2024-03-25T17:37:42.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.553+0000 7fdaf3fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdafc13f710 0x7fdafc138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45164/0 (socket says 172.21.15.96:45164) 2024-03-25T17:37:42.554 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.553+0000 7fdaf3fff640 1 -- 172.21.15.96:0/442870742 learned_addr learned my addr 172.21.15.96:0/442870742 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:42.557 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.557+0000 7fdaf8ff9640 10 client.?.objecter ms_handle_connect 0x7fdafc13f710 2024-03-25T17:37:42.558 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.557+0000 7fdaf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395342597 0 0) 0x7fdafc0abd30 con 0x7fdafc13f710 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 10 monclient(hunting): my global_id is 4617 2024-03-25T17:37:42.565 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.564+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc004eb0 con 0x7fdafc13f710 2024-03-25T17:37:42.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.567+0000 7fdaf8ff9640 10 client.?.objecter ms_handle_connect 0x7fdafc1430c0 2024-03-25T17:37:42.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.567+0000 7fdaf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:42.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.576+0000 7fdaf8ff9640 10 client.?.objecter ms_handle_connect 0x7fdafc0a98f0 2024-03-25T17:37:42.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.576+0000 7fdaf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19052509 0 0) 0x7fdadc004eb0 con 0x7fdafc13f710 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdafc0abd30 con 0x7fdafc13f710 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fdae4001a30 con 0x7fdafc13f710 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 10 monclient(hunting): dump: 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:42.579 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.578+0000 7fdaf8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:42.580 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469577439 0 0) 0x7fdafc1d77a0 con 0x7fdafc0a98f0 2024-03-25T17:37:42.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:37:42.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:42.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:42.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 10 monclient(hunting): my global_id is 4597 2024-03-25T17:37:42.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.579+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc004eb0 con 0x7fdafc0a98f0 2024-03-25T17:37:42.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3816806505 0 0) 0x7fdafc0abd30 con 0x7fdafc13f710 2024-03-25T17:37:42.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 >> v1:172.21.15.96:6790/0 conn(0x7fdafc1430c0 legacy=0x7fdafc1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:42.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 >> v1:172.21.15.203:6789/0 conn(0x7fdafc0a98f0 legacy=0x7fdafc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:42.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:37:42.584 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 1 monclient: found mon.a 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc1d83d0 con 0x7fdafc13f710 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 10 monclient: _finish_auth 0 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdaf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 5 monclient: authenticate success, global_id 4617 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 10 monclient: _renew_subs 2024-03-25T17:37:42.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc138a50 con 0x7fdafc13f710 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 10 monclient: _renew_subs 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc1e35c0 con 0x7fdafc13f710 2024-03-25T17:37:42.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.583+0000 7fdb01fc9640 1 librados: init done 2024-03-25T17:37:42.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.584+0000 7fdb01fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:42.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.584+0000 7fdb01fc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:42.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.584+0000 7fdb01fc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:42.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.584+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc1d7990 con 0x7fdafc13f710 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdae4001cb0 con 0x7fdafc13f710 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fdae4003a60 con 0x7fdafc13f710 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdaf8ff9640 10 monclient: dump: 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:42.881 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.880+0000 7fdafa7fc640 4 set_mon_vals no callback set 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.881+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fdae4003c40 con 0x7fdafc13f710 2024-03-25T17:37:42.882 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.882+0000 7fdafb7fe640 10 monclient: get_auth_request con 0x7fdadc06dd60 auth_method 0 2024-03-25T17:37:42.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.882+0000 7fdaf8ff9640 10 client.4617.objecter ms_handle_connect 0x7fdadc06dd60 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fdae4054920 con 0x7fdafc13f710 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 10 client.4617.objecter ms_dispatch 0x7fdafc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdae4050260 con 0x7fdafc13f710 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:42.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:42.887+0000 7fdaf8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:43.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.103+0000 7fdb01fc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:43.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.103+0000 7fdb01fc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:43.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.103+0000 7fdb01fc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:43.104 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.103+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdafc0627b0 con 0x7fdafc13f710 2024-03-25T17:37:43.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.124+0000 7fdaf8ff9640 1 -- 172.21.15.96:0/442870742 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fdae4052360 con 0x7fdafc13f710 2024-03-25T17:37:43.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.124+0000 7fdaf8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:43.125 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.124+0000 7fdaf8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:43.125 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:43.126 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 10 librados: watch_flush enter 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 10 librados: watch_flush exit 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 10 client.4617.objecter successfully canceled tick 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 >> v1:172.21.15.203:6800/3611964061 conn(0x7fdadc06dd60 legacy=0x7fdadc070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 10 monclient: shutdown 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:43.128 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 >> v1:172.21.15.96:6789/0 conn(0x7fdafc13f710 legacy=0x7fdafc138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:43.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 shutdown_connections 2024-03-25T17:37:43.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 >> 172.21.15.96:0/442870742 conn(0x7fdafc0b7ea0 msgr2=0x7fdafc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:43.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 shutdown_connections 2024-03-25T17:37:43.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 -- 172.21.15.96:0/442870742 wait complete. 2024-03-25T17:37:43.129 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:43.127+0000 7fdb01fc9640 1 librados: shutdown 2024-03-25T17:37:43.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:46.656 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:10.882 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.711 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:10.938 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.744 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:10.971 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.755 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:10.970 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.759 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:10.986 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.764 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:10.979 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.778 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:11.005 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.796 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:11.011 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:46.801 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:11.028 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:49.947 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:50.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.059+0000 7f128eb9d640 10 monclient: get_monmap_and_config 2024-03-25T17:37:50.059 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.059+0000 7f128eb9d640 10 monclient: build_initial_monmap 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.059+0000 7f128eb9d640 10 monclient: monmap: 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:50.060000+0000 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:50.060000+0000 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:50.060 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:50.061 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:50.061 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.062 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.060+0000 7f128eb9d640 1 Processor -- start 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.060+0000 7f128eb9d640 1 -- start start 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.060+0000 7f128eb9d640 10 monclient: init 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f128806f460 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f128806f840 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f12880724c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12880716b0 con 0x7f128806f460 2024-03-25T17:37:50.063 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1288070650 con 0x7f128806f840 2024-03-25T17:37:50.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12880af1f0 con 0x7f12880724c0 2024-03-25T17:37:50.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:50.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.061+0000 7f128eb9d640 10 monclient(hunting): authenticate will time out at 5828.342373s 2024-03-25T17:37:50.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.069+0000 7f128c939640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f128806f460 0x7f128813c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33276/0 (socket says 172.21.15.96:33276) 2024-03-25T17:37:50.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.070+0000 7f128c939640 1 -- 172.21.15.96:0/3902493921 learned_addr learned my addr 172.21.15.96:0/3902493921 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987525435 0 0) 0x7f12880716b0 con 0x7f128806f460 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 10 monclient(hunting): my global_id is 4620 2024-03-25T17:37:50.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.081+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f128806f460 2024-03-25T17:37:50.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077802885 0 0) 0x7f12880af1f0 con 0x7f12880724c0 2024-03-25T17:37:50.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.090 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T17:37:50.091 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.090+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12880716b0 con 0x7f12880724c0 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036009630 0 0) 0x7f1288070650 con 0x7f128806f840 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T17:37:50.100 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.100+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12880af1f0 con 0x7f128806f840 2024-03-25T17:37:50.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870896458 0 0) 0x7f12880716b0 con 0x7f12880724c0 2024-03-25T17:37:50.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 >> v1:172.21.15.203:6789/0 conn(0x7f128806f840 legacy=0x7f1288071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 >> v1:172.21.15.96:6789/0 conn(0x7f128806f460 legacy=0x7f128813c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:50.108 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 monclient: found mon.noname-b 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288072930 con 0x7f12880724c0 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f12800019f0 con 0x7f12880724c0 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:50.109 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f12877fe640 10 monclient: dump: 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:50.110 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.111 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f128eb9d640 5 monclient: authenticate success, global_id 4604 2024-03-25T17:37:50.111 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.108+0000 7f128eb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:50.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1280001b50 con 0x7f12880724c0 2024-03-25T17:37:50.131 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 1 -- 172.21.15.96:0/3902493921 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1280001f60 con 0x7f12880724c0 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f128eb9d640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f128eb9d640 4 set_mon_vals no callback set 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f12877fe640 10 monclient: dump: 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.132 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:50.133 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.133 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:50.133 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:50.133 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:50.133 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f128eb9d640 10 monclient: shutdown 2024-03-25T17:37:50.135 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f128eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.132+0000 7f128eb9d640 1 -- 172.21.15.96:0/3902493921 >> v1:172.21.15.96:6790/0 conn(0x7f12880724c0 legacy=0x7f128814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 1 -- 172.21.15.96:0/3902493921 shutdown_connections 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 1 -- 172.21.15.96:0/3902493921 >> 172.21.15.96:0/3902493921 conn(0x7f12880b4b80 msgr2=0x7f12880b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 1 -- 172.21.15.96:0/3902493921 shutdown_connections 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 1 -- 172.21.15.96:0/3902493921 wait complete. 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 10 monclient: build_initial_monmap 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.133+0000 7f128eb9d640 10 monclient: monmap: 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:50.136 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 librados: starting msgr at 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 librados: starting objecter 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 Processor -- start 2024-03-25T17:37:50.137 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 -- start start 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 librados: setting wanted keys 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 1 librados: calling monclient init 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.134+0000 7f128eb9d640 10 monclient: init 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f128806f460 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f128806f840 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f12880724c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12881d6600 con 0x7f128806f840 2024-03-25T17:37:50.138 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12881c8c20 con 0x7f128806f460 2024-03-25T17:37:50.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12881d6290 con 0x7f12880724c0 2024-03-25T17:37:50.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:50.139 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.135+0000 7f128eb9d640 10 monclient(hunting): authenticate will time out at 5828.416059s 2024-03-25T17:37:50.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.146+0000 7f128d13a640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f12880724c0 0x7f128813d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:57716/0 (socket says 172.21.15.96:57716) 2024-03-25T17:37:50.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.146+0000 7f128d13a640 1 -- 172.21.15.96:0/589586442 learned_addr learned my addr 172.21.15.96:0/589586442 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:50.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.154+0000 7f12857fa640 10 client.?.objecter ms_handle_connect 0x7f128806f460 2024-03-25T17:37:50.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.154+0000 7f12857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:50.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.154+0000 7f12857fa640 10 client.?.objecter ms_handle_connect 0x7f12880724c0 2024-03-25T17:37:50.154 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.154+0000 7f12857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459177734 0 0) 0x7f12881d6290 con 0x7f12880724c0 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 10 monclient(hunting): my global_id is 4607 2024-03-25T17:37:50.169 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.169+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c004eb0 con 0x7f12880724c0 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125920382 0 0) 0x7f12881c8c20 con 0x7f128806f460 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 10 monclient(hunting): my global_id is 4603 2024-03-25T17:37:50.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.171+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12881d6290 con 0x7f128806f460 2024-03-25T17:37:50.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.173+0000 7f12857fa640 10 client.?.objecter ms_handle_connect 0x7f128806f840 2024-03-25T17:37:50.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.173+0000 7f12857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:50.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064362322 0 0) 0x7f12881d6600 con 0x7f128806f840 2024-03-25T17:37:50.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:50.182 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:50.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:50.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 10 monclient(hunting): my global_id is 4623 2024-03-25T17:37:50.183 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.182+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12881c8c20 con 0x7f128806f840 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583841963 0 0) 0x7f12881d6290 con 0x7f128806f460 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12881d6600 con 0x7f128806f460 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f127c001950 con 0x7f128806f460 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:50.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 10 monclient(hunting): dump: 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.190+0000 7f12857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952228752 0 0) 0x7f126c004eb0 con 0x7f12880724c0 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12881d6290 con 0x7f12880724c0 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1280001fa0 con 0x7f12880724c0 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:50.192 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.192+0000 7f12857fa640 10 monclient(hunting): dump: 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:50.193 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2935839911 0 0) 0x7f12881d6290 con 0x7f12880724c0 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 >> v1:172.21.15.203:6789/0 conn(0x7f128806f460 legacy=0x7f12881d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 >> v1:172.21.15.96:6789/0 conn(0x7f128806f840 legacy=0x7f128813ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 10 monclient: _finish_hunting 0 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 1 monclient: found mon.c 2024-03-25T17:37:50.214 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12880ae2c0 con 0x7f12880724c0 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 10 monclient: _finish_auth 0 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 10 monclient: _check_auth_tickets 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f12857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 5 monclient: authenticate success, global_id 4607 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 10 monclient: _renew_subs 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12880ae510 con 0x7f12880724c0 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:50.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 10 monclient: _renew_subs 2024-03-25T17:37:50.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f128813fca0 con 0x7f12880724c0 2024-03-25T17:37:50.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.213+0000 7f128eb9d640 1 librados: init done 2024-03-25T17:37:50.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.215+0000 7f128eb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:50.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.215+0000 7f128eb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:50.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.215+0000 7f128eb9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.215+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12881d6420 con 0x7f12880724c0 2024-03-25T17:37:50.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1280003c20 con 0x7f12880724c0 2024-03-25T17:37:50.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:50.566 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1280003f70 con 0x7f12880724c0 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f12857fa640 10 monclient: dump: 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:50.567 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.566+0000 7f1286ffd640 4 set_mon_vals no callback set 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.567+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1280004190 con 0x7f12880724c0 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.568+0000 7f128c939640 10 monclient: get_auth_request con 0x7f126c06df10 auth_method 0 2024-03-25T17:37:50.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.568+0000 7f12857fa640 10 client.4607.objecter ms_handle_connect 0x7f126c06df10 2024-03-25T17:37:50.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f1280054bc0 con 0x7f12880724c0 2024-03-25T17:37:50.572 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 10 client.4607.objecter ms_dispatch 0x7f1288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12800506d0 con 0x7f12880724c0 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:50.573 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.572+0000 7f12857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:50.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.817+0000 7f128eb9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:50.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.817+0000 7f128eb9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:50.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.817+0000 7f128eb9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:37:50.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.817+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12880627b0 con 0x7f12880724c0 2024-03-25T17:37:50.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.844+0000 7f12857fa640 1 -- 172.21.15.96:0/589586442 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f1280052600 con 0x7f12880724c0 2024-03-25T17:37:50.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.844+0000 7f12857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:50.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.844+0000 7f12857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:50.845 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:50.845 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:50.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 10 librados: watch_flush enter 2024-03-25T17:37:50.847 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 10 librados: watch_flush exit 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 10 client.4607.objecter successfully canceled tick 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 >> v1:172.21.15.203:6800/3611964061 conn(0x7f126c06df10 legacy=0x7f126c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 10 monclient: shutdown 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 >> v1:172.21.15.96:6790/0 conn(0x7f12880724c0 legacy=0x7f128813d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 shutdown_connections 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 >> 172.21.15.96:0/589586442 conn(0x7f12880b4b80 msgr2=0x7f12880b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 shutdown_connections 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 -- 172.21.15.96:0/589586442 wait complete. 2024-03-25T17:37:50.848 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:50.847+0000 7f128eb9d640 1 librados: shutdown 2024-03-25T17:37:50.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:57.668 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.783+0000 7f1f5da2e640 10 monclient: get_monmap_and_config 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.784+0000 7f1f5da2e640 10 monclient: build_initial_monmap 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.784+0000 7f1f5da2e640 10 monclient: monmap: 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:37:57.785300+0000 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:37:57.785300+0000 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:57.785 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:57.786 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:57.786 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:37:57.786 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:37:57.786 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.784+0000 7f1f5da2e640 1 Processor -- start 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.784+0000 7f1f5da2e640 1 -- start start 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.784+0000 7f1f5da2e640 10 monclient: init 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.785+0000 7f1f5da2e640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.785+0000 7f1f5da2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.785+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f580a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:57.788 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f5813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f581430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f58142520 con 0x7f1f580a98f0 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580b28a0 con 0x7f1f5813f710 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580b2a30 con 0x7f1f581430c0 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:57.789 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.786+0000 7f1f5da2e640 10 monclient(hunting): authenticate will time out at 5836.068078s 2024-03-25T17:37:57.791 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.791+0000 7f1f577fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f1f581430c0 0x7f1f5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:53008/0 (socket says 172.21.15.96:53008) 2024-03-25T17:37:57.792 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.791+0000 7f1f577fe640 1 -- 172.21.15.96:0/2222146195 learned_addr learned my addr 172.21.15.96:0/2222146195 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414595760 0 0) 0x7f1f58142520 con 0x7f1f580a98f0 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 10 monclient(hunting): my global_id is 4626 2024-03-25T17:37:57.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.802+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40003520 con 0x7f1f580a98f0 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619812327 0 0) 0x7f1f580b28a0 con 0x7f1f5813f710 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 10 monclient(hunting): my global_id is 4606 2024-03-25T17:37:57.808 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.807+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f58142520 con 0x7f1f5813f710 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498836815 0 0) 0x7f1f580b2a30 con 0x7f1f581430c0 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 10 monclient(hunting): my global_id is 4610 2024-03-25T17:37:57.809 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.808+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f580b28a0 con 0x7f1f581430c0 2024-03-25T17:37:57.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 305974935 0 0) 0x7f1f40003520 con 0x7f1f580a98f0 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 >> v1:172.21.15.96:6790/0 conn(0x7f1f581430c0 legacy=0x7f1f5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 >> v1:172.21.15.203:6789/0 conn(0x7f1f5813f710 legacy=0x7f1f581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 monclient: found mon.noname-a 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580b44c0 con 0x7f1f580a98f0 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:57.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1f48001690 con 0x7f1f580a98f0 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f5da2e640 5 monclient: authenticate success, global_id 4626 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f5da2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f55ffb640 10 monclient: dump: 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.817 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:57.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.814+0000 7f1f5da2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:57.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f480017f0 con 0x7f1f580a98f0 2024-03-25T17:37:57.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:57.820 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 1 -- 172.21.15.96:0/2222146195 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1f48001c00 con 0x7f1f580a98f0 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f55ffb640 10 monclient: dump: 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:57.821 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:57.822 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:57.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f5da2e640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:57.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.819+0000 7f1f5da2e640 4 set_mon_vals no callback set 2024-03-25T17:37:57.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 10 monclient: shutdown 2024-03-25T17:37:57.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:57.824 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 1 -- 172.21.15.96:0/2222146195 >> v1:172.21.15.96:6789/0 conn(0x7f1f580a98f0 legacy=0x7f1f580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 1 -- 172.21.15.96:0/2222146195 shutdown_connections 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 1 -- 172.21.15.96:0/2222146195 >> 172.21.15.96:0/2222146195 conn(0x7f1f580b7ea0 msgr2=0x7f1f580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 1 -- 172.21.15.96:0/2222146195 shutdown_connections 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.820+0000 7f1f5da2e640 1 -- 172.21.15.96:0/2222146195 wait complete. 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.821+0000 7f1f5da2e640 10 monclient: build_initial_monmap 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.821+0000 7f1f5da2e640 10 monclient: monmap: 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:37:57.825 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 librados: starting msgr at 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 librados: starting objecter 2024-03-25T17:37:57.826 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 Processor -- start 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 -- start start 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 librados: setting wanted keys 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 1 librados: calling monclient init 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 10 monclient: init 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f580a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.822+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f5813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f581430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:37:57.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f581db2f0 con 0x7f1f5813f710 2024-03-25T17:37:57.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f58139180 con 0x7f1f580a98f0 2024-03-25T17:37:57.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f58137670 con 0x7f1f581430c0 2024-03-25T17:37:57.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:57.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.823+0000 7f1f5da2e640 10 monclient(hunting): authenticate will time out at 5836.105058s 2024-03-25T17:37:57.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.836+0000 7f1f567fc640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1f5813f710 0x7f1f58132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40834/0 (socket says 172.21.15.96:40834) 2024-03-25T17:37:57.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.836+0000 7f1f567fc640 1 -- 172.21.15.96:0/3913072585 learned_addr learned my addr 172.21.15.96:0/3913072585 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:37:57.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.836+0000 7f1f3ffff640 10 client.?.objecter ms_handle_connect 0x7f1f5813f710 2024-03-25T17:37:57.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.836+0000 7f1f3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:57.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.844+0000 7f1f3ffff640 10 client.?.objecter ms_handle_connect 0x7f1f580a98f0 2024-03-25T17:37:57.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.844+0000 7f1f3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:57.845 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.845+0000 7f1f3ffff640 10 client.?.objecter ms_handle_connect 0x7f1f581430c0 2024-03-25T17:37:57.846 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.845+0000 7f1f3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389352840 0 0) 0x7f1f58139180 con 0x7f1f580a98f0 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 10 monclient(hunting): my global_id is 4609 2024-03-25T17:37:57.854 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.853+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f44004eb0 con 0x7f1f580a98f0 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526551947 0 0) 0x7f1f581db2f0 con 0x7f1f5813f710 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 10 monclient(hunting): my global_id is 4629 2024-03-25T17:37:57.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.861+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f58139180 con 0x7f1f5813f710 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415914100 0 0) 0x7f1f58137670 con 0x7f1f581430c0 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 10 monclient(hunting): my global_id is 4613 2024-03-25T17:37:57.870 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.869+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f581db2f0 con 0x7f1f581430c0 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.883+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417053180 0 0) 0x7f1f58139180 con 0x7f1f5813f710 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.883+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f58137670 con 0x7f1f5813f710 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.883+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1f4c0019b0 con 0x7f1f5813f710 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.883+0000 7f1f3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.884+0000 7f1f3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.884+0000 7f1f3ffff640 10 monclient(hunting): dump: 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:57.885 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:57.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.884+0000 7f1f3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.957+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3215719437 0 0) 0x7f1f58137670 con 0x7f1f5813f710 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.957+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 >> v1:172.21.15.96:6790/0 conn(0x7f1f581430c0 legacy=0x7f1f581e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.957+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 >> v1:172.21.15.203:6789/0 conn(0x7f1f580a98f0 legacy=0x7f1f581dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.957+0000 7f1f3ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.957+0000 7f1f3ffff640 1 monclient: found mon.a 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58137200 con 0x7f1f5813f710 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 10 monclient: _finish_auth 0 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:57.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 5 monclient: authenticate success, global_id 4629 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 10 monclient: _renew_subs 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f580b7290 con 0x7f1f5813f710 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 10 monclient: _renew_subs 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f58133680 con 0x7f1f5813f710 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.958+0000 7f1f5da2e640 1 librados: init done 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.959+0000 7f1f5da2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.959+0000 7f1f5da2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:57.961 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.959+0000 7f1f5da2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:57.962 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:57.959+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f580627b0 con 0x7f1f5813f710 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f4c003610 con 0x7f1f5813f710 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1f4c0039a0 con 0x7f1f5813f710 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f3ffff640 10 monclient: dump: 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:37:58.280 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.278+0000 7f1f557fa640 4 set_mon_vals no callback set 2024-03-25T17:37:58.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.279+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1f4c003bc0 con 0x7f1f5813f710 2024-03-25T17:37:58.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.280+0000 7f1f56ffd640 10 monclient: get_auth_request con 0x7f1f4406de20 auth_method 0 2024-03-25T17:37:58.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.280+0000 7f1f3ffff640 10 client.4629.objecter ms_handle_connect 0x7f1f4406de20 2024-03-25T17:37:58.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f1f4c054860 con 0x7f1f5813f710 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 10 client.4629.objecter ms_dispatch 0x7f1f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f4c050370 con 0x7f1f5813f710 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:58.287 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.285+0000 7f1f3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:58.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.538+0000 7f1f5da2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:58.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.538+0000 7f1f5da2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:58.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.538+0000 7f1f5da2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:37:58.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.538+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f5813e2b0 con 0x7f1f5813f710 2024-03-25T17:37:58.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.590+0000 7f1f3ffff640 1 -- 172.21.15.96:0/3913072585 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f1f4c0522a0 con 0x7f1f5813f710 2024-03-25T17:37:58.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.590+0000 7f1f3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:58.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.590+0000 7f1f3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:58.592 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:37:58.592 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:58.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 10 librados: watch_flush enter 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 10 librados: watch_flush exit 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 10 client.4629.objecter successfully canceled tick 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 >> v1:172.21.15.203:6800/3611964061 conn(0x7f1f4406de20 legacy=0x7f1f440702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 10 monclient: shutdown 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 >> v1:172.21.15.96:6789/0 conn(0x7f1f5813f710 legacy=0x7f1f58132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 shutdown_connections 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 >> 172.21.15.96:0/3913072585 conn(0x7f1f580b7ea0 msgr2=0x7f1f58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.593+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 shutdown_connections 2024-03-25T17:37:58.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.594+0000 7f1f5da2e640 1 -- 172.21.15.96:0/3913072585 wait complete. 2024-03-25T17:37:58.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:37:58.594+0000 7f1f5da2e640 1 librados: shutdown 2024-03-25T17:37:58.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:05.415 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.534+0000 7f7f540d6640 10 monclient: get_monmap_and_config 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.534+0000 7f7f540d6640 10 monclient: build_initial_monmap 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.534+0000 7f7f540d6640 10 monclient: monmap: 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:05.535854+0000 2024-03-25T17:38:05.535 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:05.535854+0000 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.535+0000 7f7f540d6640 1 Processor -- start 2024-03-25T17:38:05.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.535+0000 7f7f540d6640 1 -- start start 2024-03-25T17:38:05.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.535+0000 7f7f540d6640 10 monclient: init 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c06e200 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c072730 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c06e5e0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0ae470 con 0x7f7f4c06e200 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0b0320 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c06efa0 con 0x7f7f4c072730 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:05.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.536+0000 7f7f540d6640 10 monclient(hunting): authenticate will time out at 5843.818769s 2024-03-25T17:38:05.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.542+0000 7f7f52673640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7f4c06e5e0 0x7f7f4c152080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:51818/0 (socket says 172.21.15.96:51818) 2024-03-25T17:38:05.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.542+0000 7f7f52673640 1 -- 172.21.15.96:0/3381543542 learned_addr learned my addr 172.21.15.96:0/3381543542 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:05.568 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.567+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674603683 0 0) 0x7f7f4c06efa0 con 0x7f7f4c072730 2024-03-25T17:38:05.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.568+0000 7f7f50e70640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.568+0000 7f7f50e70640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.568+0000 7f7f50e70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.568+0000 7f7f50e70640 10 monclient(hunting): my global_id is 4616 2024-03-25T17:38:05.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.568+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38003520 con 0x7f7f4c072730 2024-03-25T17:38:05.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682539525 0 0) 0x7f7f4c0ae470 con 0x7f7f4c06e200 2024-03-25T17:38:05.577 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 10 monclient(hunting): my global_id is 4632 2024-03-25T17:38:05.578 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.577+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c06efa0 con 0x7f7f4c06e200 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793501028 0 0) 0x7f7f4c0b0320 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 10 monclient(hunting): my global_id is 4612 2024-03-25T17:38:05.581 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.580+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c0ae470 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.584+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564874524 0 0) 0x7f7f38003520 con 0x7f7f4c072730 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.584+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 >> v1:172.21.15.203:6789/0 conn(0x7f7f4c06e5e0 legacy=0x7f7f4c152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.584+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 >> v1:172.21.15.96:6789/0 conn(0x7f7f4c06e200 legacy=0x7f7f4c072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.584+0000 7f7f50e70640 10 monclient: _finish_hunting 0 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.584+0000 7f7f50e70640 1 monclient: found mon.noname-b 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c06f3f0 con 0x7f7f4c072730 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: _finish_auth 0 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: _check_auth_tickets 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f540d6640 5 monclient: authenticate success, global_id 4616 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f540d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f3c001690 con 0x7f7f4c072730 2024-03-25T17:38:05.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f50e70640 10 monclient: dump: 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.587 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.585+0000 7f7f540d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c0017f0 con 0x7f7f4c072730 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f540d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f540d6640 4 set_mon_vals no callback set 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 1 -- 172.21.15.96:0/3381543542 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f3c001c00 con 0x7f7f4c072730 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.608+0000 7f7f50e70640 10 monclient: dump: 2024-03-25T17:38:05.609 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.610 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.609+0000 7f7f540d6640 10 monclient: shutdown 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.609+0000 7f7f540d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.609+0000 7f7f540d6640 1 -- 172.21.15.96:0/3381543542 >> v1:172.21.15.96:6790/0 conn(0x7f7f4c072730 legacy=0x7f7f4c146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.609+0000 7f7f540d6640 1 -- 172.21.15.96:0/3381543542 shutdown_connections 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.609+0000 7f7f540d6640 1 -- 172.21.15.96:0/3381543542 >> 172.21.15.96:0/3381543542 conn(0x7f7f4c06b630 msgr2=0x7f7f4c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.610+0000 7f7f540d6640 1 -- 172.21.15.96:0/3381543542 shutdown_connections 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.610+0000 7f7f540d6640 1 -- 172.21.15.96:0/3381543542 wait complete. 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.610+0000 7f7f540d6640 10 monclient: build_initial_monmap 2024-03-25T17:38:05.611 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.610+0000 7f7f540d6640 10 monclient: monmap: 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:05.612 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 librados: starting msgr at 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 librados: starting objecter 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 Processor -- start 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 -- start start 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 librados: setting wanted keys 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 1 librados: calling monclient init 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.611+0000 7f7f540d6640 10 monclient: init 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c06e200 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:05.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c06e5e0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c072730 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c137ed0 con 0x7f7f4c072730 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c145c20 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c14a4e0 con 0x7f7f4c06e200 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:05.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.612+0000 7f7f540d6640 10 monclient(hunting): authenticate will time out at 5843.894183s 2024-03-25T17:38:05.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.621+0000 7f7f51671640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7f4c06e5e0 0x7f7f4c1370b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:51880/0 (socket says 172.21.15.96:51880) 2024-03-25T17:38:05.622 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.621+0000 7f7f51671640 1 -- 172.21.15.96:0/553589790 learned_addr learned my addr 172.21.15.96:0/553589790 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:05.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.625+0000 7f7f427fc640 10 client.?.objecter ms_handle_connect 0x7f7f4c06e5e0 2024-03-25T17:38:05.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.625+0000 7f7f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:05.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.627+0000 7f7f427fc640 10 client.?.objecter ms_handle_connect 0x7f7f4c072730 2024-03-25T17:38:05.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.627+0000 7f7f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:05.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.629+0000 7f7f427fc640 10 client.?.objecter ms_handle_connect 0x7f7f4c06e200 2024-03-25T17:38:05.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.629+0000 7f7f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:05.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690332807 0 0) 0x7f7f4c145c20 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 10 monclient(hunting): my global_id is 4615 2024-03-25T17:38:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.635+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30004eb0 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881707466 0 0) 0x7f7f4c137ed0 con 0x7f7f4c072730 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 10 monclient(hunting): my global_id is 4635 2024-03-25T17:38:05.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.640+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c145c20 con 0x7f7f4c072730 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133934924 0 0) 0x7f7f4c14a4e0 con 0x7f7f4c06e200 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T17:38:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.641+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c137ed0 con 0x7f7f4c06e200 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180765214 0 0) 0x7f7f4c137ed0 con 0x7f7f4c06e200 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c14a4e0 con 0x7f7f4c06e200 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f480018f0 con 0x7f7f4c06e200 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 10 monclient(hunting): dump: 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.649 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.650 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.648+0000 7f7f427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:05.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4221806252 0 0) 0x7f7f4c145c20 con 0x7f7f4c072730 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c137ed0 con 0x7f7f4c072730 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f44001bf0 con 0x7f7f4c072730 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.651+0000 7f7f427fc640 10 monclient(hunting): dump: 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.652 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856717456 0 0) 0x7f7f30004eb0 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c145c20 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f3c001bc0 con 0x7f7f4c06e5e0 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.653+0000 7f7f427fc640 10 monclient(hunting): dump: 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.654 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.655 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.655 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.655 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.655 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.663+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1674763455 0 0) 0x7f7f4c137ed0 con 0x7f7f4c072730 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 >> v1:172.21.15.96:6790/0 conn(0x7f7f4c06e200 legacy=0x7f7f4c148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 >> v1:172.21.15.203:6789/0 conn(0x7f7f4c06e5e0 legacy=0x7f7f4c1370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 10 monclient: _finish_hunting 0 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 1 monclient: found mon.a 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c14a7c0 con 0x7f7f4c072730 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 10 monclient: _finish_auth 0 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 10 monclient: _check_auth_tickets 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 5 monclient: authenticate success, global_id 4635 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 10 monclient: _renew_subs 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:05.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c1381c0 con 0x7f7f4c072730 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 10 monclient: _renew_subs 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c0a9650 con 0x7f7f4c072730 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.664+0000 7f7f540d6640 1 librados: init done 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.665+0000 7f7f540d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.665+0000 7f7f540d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.665+0000 7f7f540d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:05.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.665+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c145a30 con 0x7f7f4c072730 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f44002080 con 0x7f7f4c072730 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7f44004d80 con 0x7f7f4c072730 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f427fc640 10 monclient: dump: 2024-03-25T17:38:05.967 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:05.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.966+0000 7f7f43fff640 4 set_mon_vals no callback set 2024-03-25T17:38:05.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.967+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7f4401ad90 con 0x7f7f4c072730 2024-03-25T17:38:05.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.967+0000 7f7f51e72640 10 monclient: get_auth_request con 0x7f7f3006e110 auth_method 0 2024-03-25T17:38:05.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.968+0000 7f7f427fc640 10 client.4635.objecter ms_handle_connect 0x7f7f3006e110 2024-03-25T17:38:05.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f7f44054a10 con 0x7f7f4c072730 2024-03-25T17:38:05.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 10 client.4635.objecter ms_dispatch 0x7f7f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:05.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:05.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:05.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T17:38:05.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f44050520 con 0x7f7f4c072730 2024-03-25T17:38:05.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:05.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:05.974+0000 7f7f427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:06.185 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.184+0000 7f7f540d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:06.185 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.184+0000 7f7f540d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:06.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.184+0000 7f7f540d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:06.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.184+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f4c0627b0 con 0x7f7f4c072730 2024-03-25T17:38:06.215 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.215+0000 7f7f427fc640 1 -- 172.21.15.96:0/553589790 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f7f44052450 con 0x7f7f4c072730 2024-03-25T17:38:06.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.215+0000 7f7f427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:06.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.215+0000 7f7f427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:06.216 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:06.216 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:06.218 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 10 librados: watch_flush enter 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 10 librados: watch_flush exit 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 10 client.4635.objecter successfully canceled tick 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 >> v1:172.21.15.203:6800/3611964061 conn(0x7f7f3006e110 legacy=0x7f7f300705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 10 monclient: shutdown 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 >> v1:172.21.15.96:6789/0 conn(0x7f7f4c072730 legacy=0x7f7f4c1377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 shutdown_connections 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 >> 172.21.15.96:0/553589790 conn(0x7f7f4c06b630 msgr2=0x7f7f4c0aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 shutdown_connections 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 -- 172.21.15.96:0/553589790 wait complete. 2024-03-25T17:38:06.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:06.217+0000 7f7f540d6640 1 librados: shutdown 2024-03-25T17:38:06.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:06.762 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:30.989 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.773 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:30.999 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.798 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:31.024 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.825 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:31.040 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.829 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:31.044 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.839 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:31.066 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.844 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:31.071 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.870 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:31.085 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:06.876 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:31.103 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:13.036 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.152+0000 7f5f9efb5640 10 monclient: get_monmap_and_config 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.152+0000 7f5f9efb5640 10 monclient: build_initial_monmap 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.152+0000 7f5f9efb5640 10 monclient: monmap: 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.153 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:13.153726+0000 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:13.153726+0000 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:13.154 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.153+0000 7f5f9efb5640 1 Processor -- start 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.153+0000 7f5f9efb5640 1 -- start start 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.153+0000 7f5f9efb5640 10 monclient: init 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-c con 0x7f5f980a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:13.155 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-a con 0x7f5f981430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98142520 con 0x7f5f981430c0 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b28a0 con 0x7f5f980a98f0 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b2a30 con 0x7f5f9813f710 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:13.156 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.154+0000 7f5f9efb5640 10 monclient(hunting): authenticate will time out at 5851.436084s 2024-03-25T17:38:13.160 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.159+0000 7f5f9cd51640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5f980a98f0 0x7f5f980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:56504/0 (socket says 172.21.15.96:56504) 2024-03-25T17:38:13.160 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.159+0000 7f5f9cd51640 1 -- 172.21.15.96:0/1440674478 learned_addr learned my addr 172.21.15.96:0/1440674478 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630750586 0 0) 0x7f5f980b28a0 con 0x7f5f980a98f0 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 10 monclient(hunting): my global_id is 4618 2024-03-25T17:38:13.166 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.165+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88003520 con 0x7f5f980a98f0 2024-03-25T17:38:13.172 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228765558 0 0) 0x7f5f98142520 con 0x7f5f981430c0 2024-03-25T17:38:13.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 10 monclient(hunting): my global_id is 4638 2024-03-25T17:38:13.173 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.172+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f980b28a0 con 0x7f5f981430c0 2024-03-25T17:38:13.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057260087 0 0) 0x7f5f980b2a30 con 0x7f5f9813f710 2024-03-25T17:38:13.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.186 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 10 monclient(hunting): my global_id is 4622 2024-03-25T17:38:13.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.186+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98142520 con 0x7f5f9813f710 2024-03-25T17:38:13.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520128311 0 0) 0x7f5f88003520 con 0x7f5f980a98f0 2024-03-25T17:38:13.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 >> v1:172.21.15.96:6790/0 conn(0x7f5f9813f710 legacy=0x7f5f981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 >> v1:172.21.15.96:6789/0 conn(0x7f5f981430c0 legacy=0x7f5f9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: _finish_hunting 0 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 monclient: found mon.noname-c 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980b44c0 con 0x7f5f980a98f0 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: _finish_auth 0 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: _check_auth_tickets 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5f8c001690 con 0x7f5f980a98f0 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f977fe640 10 monclient: dump: 2024-03-25T17:38:13.190 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f9efb5640 5 monclient: authenticate success, global_id 4618 2024-03-25T17:38:13.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.188+0000 7f5f9efb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f8c0017f0 con 0x7f5f980a98f0 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 1 -- 172.21.15.96:0/1440674478 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5f8c001c00 con 0x7f5f980a98f0 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f9efb5640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f9efb5640 4 set_mon_vals no callback set 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.207+0000 7f5f977fe640 10 monclient: dump: 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:13.208 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:13.209 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 10 monclient: shutdown 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 1 -- 172.21.15.96:0/1440674478 >> v1:172.21.15.203:6789/0 conn(0x7f5f980a98f0 legacy=0x7f5f980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 1 -- 172.21.15.96:0/1440674478 shutdown_connections 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 1 -- 172.21.15.96:0/1440674478 >> 172.21.15.96:0/1440674478 conn(0x7f5f980b7ea0 msgr2=0x7f5f980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 1 -- 172.21.15.96:0/1440674478 shutdown_connections 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.208+0000 7f5f9efb5640 1 -- 172.21.15.96:0/1440674478 wait complete. 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 10 monclient: build_initial_monmap 2024-03-25T17:38:13.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 10 monclient: monmap: 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 librados: starting msgr at 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 librados: starting objecter 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 Processor -- start 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 -- start start 2024-03-25T17:38:13.212 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 librados: setting wanted keys 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 1 librados: calling monclient init 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.209+0000 7f5f9efb5640 10 monclient: init 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-b con 0x7f5f980a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient(hunting): picked mon.noname-a con 0x7f5f981430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98139a40 con 0x7f5f981430c0 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98133f20 con 0x7f5f980a98f0 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980ab950 con 0x7f5f9813f710 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:13.213 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.210+0000 7f5f9efb5640 10 monclient(hunting): authenticate will time out at 5851.492597s 2024-03-25T17:38:13.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.220+0000 7f5f9d552640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5f981430c0 0x7f5f98133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59580/0 (socket says 172.21.15.96:59580) 2024-03-25T17:38:13.221 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.220+0000 7f5f9d552640 1 -- 172.21.15.96:0/419913896 learned_addr learned my addr 172.21.15.96:0/419913896 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:13.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.221+0000 7f5f957fa640 10 client.?.objecter ms_handle_connect 0x7f5f981430c0 2024-03-25T17:38:13.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.221+0000 7f5f957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:13.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.230+0000 7f5f957fa640 10 client.?.objecter ms_handle_connect 0x7f5f9813f710 2024-03-25T17:38:13.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.230+0000 7f5f957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:13.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461957945 0 0) 0x7f5f98139a40 con 0x7f5f981430c0 2024-03-25T17:38:13.232 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 10 monclient(hunting): my global_id is 4641 2024-03-25T17:38:13.233 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.231+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f7c004eb0 con 0x7f5f981430c0 2024-03-25T17:38:13.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.234+0000 7f5f957fa640 10 client.?.objecter ms_handle_connect 0x7f5f980a98f0 2024-03-25T17:38:13.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.234+0000 7f5f957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061983061 0 0) 0x7f5f98133f20 con 0x7f5f980a98f0 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): my global_id is 4621 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98139a40 con 0x7f5f980a98f0 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726624276 0 0) 0x7f5f980ab950 con 0x7f5f9813f710 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _negotiate 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 10 monclient(hunting): my global_id is 4625 2024-03-25T17:38:13.242 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.241+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98133f20 con 0x7f5f9813f710 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 974340987 0 0) 0x7f5f7c004eb0 con 0x7f5f981430c0 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f980ab950 con 0x7f5f981430c0 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5f90001c10 con 0x7f5f981430c0 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 10 monclient(hunting): dump: 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:13.248 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:13.249 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:13.249 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.247+0000 7f5f957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:13.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.252+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399030039 0 0) 0x7f5f98139a40 con 0x7f5f980a98f0 2024-03-25T17:38:13.253 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.252+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f7c004eb0 con 0x7f5f980a98f0 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.252+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5f8c001b00 con 0x7f5f980a98f0 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.252+0000 7f5f957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.253+0000 7f5f957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.253+0000 7f5f957fa640 10 monclient(hunting): dump: 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:13.254 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1435783806 0 0) 0x7f5f7c004eb0 con 0x7f5f980a98f0 2024-03-25T17:38:13.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 >> v1:172.21.15.96:6790/0 conn(0x7f5f9813f710 legacy=0x7f5f98133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 >> v1:172.21.15.96:6789/0 conn(0x7f5f981430c0 legacy=0x7f5f98133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 10 monclient: _finish_hunting 0 2024-03-25T17:38:13.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 1 monclient: found mon.b 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980abc30 con 0x7f5f980a98f0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 10 monclient: _finish_auth 0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 10 monclient: _check_auth_tickets 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 5 monclient: authenticate success, global_id 4621 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 10 monclient: _renew_subs 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f980abf10 con 0x7f5f980a98f0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 10 monclient: _renew_subs 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f98139370 con 0x7f5f980a98f0 2024-03-25T17:38:13.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.259+0000 7f5f9efb5640 1 librados: init done 2024-03-25T17:38:13.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.260+0000 7f5f9efb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:13.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.260+0000 7f5f9efb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:13.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.260+0000 7f5f9efb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.260+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f980627b0 con 0x7f5f980a98f0 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f8c004a90 con 0x7f5f980a98f0 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f5f8c001cc0 con 0x7f5f980a98f0 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f957fa640 10 monclient: dump: 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.281 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:13.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.280+0000 7f5f96ffd640 4 set_mon_vals no callback set 2024-03-25T17:38:13.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.281+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f5f8c01ac10 con 0x7f5f980a98f0 2024-03-25T17:38:13.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.282+0000 7f5f97fff640 10 monclient: get_auth_request con 0x7f5f7c06df70 auth_method 0 2024-03-25T17:38:13.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.282+0000 7f5f957fa640 10 client.4621.objecter ms_handle_connect 0x7f5f7c06df70 2024-03-25T17:38:13.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.284+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f5f8c054850 con 0x7f5f980a98f0 2024-03-25T17:38:13.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.284+0000 7f5f957fa640 10 client.4621.objecter ms_dispatch 0x7f5f98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:13.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.284+0000 7f5f957fa640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:13.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.284+0000 7f5f957fa640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:13.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.284+0000 7f5f957fa640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T17:38:13.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.555+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f8c050190 con 0x7f5f980a98f0 2024-03-25T17:38:13.555 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.555+0000 7f5f957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:13.556 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.555+0000 7f5f957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:13.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.810+0000 7f5f9efb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:13.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.810+0000 7f5f9efb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:13.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.810+0000 7f5f9efb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:13.811 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.810+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f980b7180 con 0x7f5f980a98f0 2024-03-25T17:38:13.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.839+0000 7f5f957fa640 1 -- 172.21.15.96:0/419913896 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f5f8c052290 con 0x7f5f980a98f0 2024-03-25T17:38:13.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.839+0000 7f5f957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:13.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.839+0000 7f5f957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:13.840 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:13.840 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.841+0000 7f5f9efb5640 10 librados: watch_flush enter 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.841+0000 7f5f9efb5640 10 librados: watch_flush exit 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.841+0000 7f5f9efb5640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.841+0000 7f5f9efb5640 10 client.4621.objecter successfully canceled tick 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 >> v1:172.21.15.203:6800/3611964061 conn(0x7f5f7c06df70 legacy=0x7f5f7c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 10 monclient: shutdown 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 >> v1:172.21.15.203:6789/0 conn(0x7f5f980a98f0 legacy=0x7f5f98138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 shutdown_connections 2024-03-25T17:38:13.843 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 >> 172.21.15.96:0/419913896 conn(0x7f5f980b7ea0 msgr2=0x7f5f98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:13.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 shutdown_connections 2024-03-25T17:38:13.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 -- 172.21.15.96:0/419913896 wait complete. 2024-03-25T17:38:13.844 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:13.842+0000 7f5f9efb5640 1 librados: shutdown 2024-03-25T17:38:13.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:20.661 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:20.774 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.774+0000 7f71b8cae640 10 monclient: get_monmap_and_config 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.774+0000 7f71b8cae640 10 monclient: build_initial_monmap 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.774+0000 7f71b8cae640 10 monclient: monmap: 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:20.775651+0000 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:20.775651+0000 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:20.775 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.775+0000 7f71b8cae640 1 Processor -- start 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.775+0000 7f71b8cae640 1 -- start start 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.775+0000 7f71b8cae640 10 monclient: init 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-c con 0x7f71b40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-b con 0x7f71b413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-a con 0x7f71b41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4142520 con 0x7f71b41430c0 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40b28a0 con 0x7f71b40a98f0 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40b2a30 con 0x7f71b413f710 2024-03-25T17:38:20.777 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:20.778 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.776+0000 7f71b8cae640 10 monclient(hunting): authenticate will time out at 5859.058088s 2024-03-25T17:38:20.790 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.790+0000 7f71b259c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f71b40a98f0 0x7f71b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:49472/0 (socket says 172.21.15.96:49472) 2024-03-25T17:38:20.790 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.790+0000 7f71b259c640 1 -- 172.21.15.96:0/2383924263 learned_addr learned my addr 172.21.15.96:0/2383924263 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790331629 0 0) 0x7f71b40b28a0 con 0x7f71b40a98f0 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 10 monclient(hunting): my global_id is 4624 2024-03-25T17:38:20.807 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.805+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0003520 con 0x7f71b40a98f0 2024-03-25T17:38:20.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557252306 0 0) 0x7f71a0003520 con 0x7f71b40a98f0 2024-03-25T17:38:20.816 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 >> v1:172.21.15.96:6790/0 conn(0x7f71b413f710 legacy=0x7f71b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 >> v1:172.21.15.96:6789/0 conn(0x7f71b41430c0 legacy=0x7f71b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 monclient: found mon.noname-c 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40b44c0 con 0x7f71b40a98f0 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: _finish_auth 0 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b8cae640 5 monclient: authenticate success, global_id 4624 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f71a4001690 con 0x7f71b40a98f0 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b8cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:20.817 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b1d9b640 10 monclient: dump: 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:20.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.815+0000 7f71b8cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71a40017f0 con 0x7f71b40a98f0 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 1 -- 172.21.15.96:0/2383924263 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f71a4001c00 con 0x7f71b40a98f0 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b1d9b640 10 monclient: dump: 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:20.832 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b8cae640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:20.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.831+0000 7f71b8cae640 4 set_mon_vals no callback set 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 10 monclient: shutdown 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 1 -- 172.21.15.96:0/2383924263 >> v1:172.21.15.203:6789/0 conn(0x7f71b40a98f0 legacy=0x7f71b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 1 -- 172.21.15.96:0/2383924263 shutdown_connections 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 1 -- 172.21.15.96:0/2383924263 >> 172.21.15.96:0/2383924263 conn(0x7f71b40b7ea0 msgr2=0x7f71b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 1 -- 172.21.15.96:0/2383924263 shutdown_connections 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.832+0000 7f71b8cae640 1 -- 172.21.15.96:0/2383924263 wait complete. 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.833+0000 7f71b8cae640 10 monclient: build_initial_monmap 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.833+0000 7f71b8cae640 10 monclient: monmap: 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:20.835 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 librados: starting msgr at 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 librados: starting objecter 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 Processor -- start 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 -- start start 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 librados: setting wanted keys 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 1 librados: calling monclient init 2024-03-25T17:38:20.836 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.834+0000 7f71b8cae640 10 monclient: init 2024-03-25T17:38:20.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-a con 0x7f71b40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-c con 0x7f71b413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient(hunting): picked mon.noname-b con 0x7f71b41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4138b20 con 0x7f71b40a98f0 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4137a80 con 0x7f71b41430c0 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4139880 con 0x7f71b413f710 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:20.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.835+0000 7f71b8cae640 10 monclient(hunting): authenticate will time out at 5859.117083s 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.837+0000 7f71b2d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f71b41430c0 0x7f71b41e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:49530/0 (socket says 172.21.15.96:49530) 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.837+0000 7f71b2d9d640 1 -- 172.21.15.96:0/3362650516 learned_addr learned my addr 172.21.15.96:0/3362650516 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.838+0000 7f71aaffd640 10 client.?.objecter ms_handle_connect 0x7f71b41430c0 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.838+0000 7f71aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.838+0000 7f71aaffd640 10 client.?.objecter ms_handle_connect 0x7f71b413f710 2024-03-25T17:38:20.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.838+0000 7f71aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:20.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.898+0000 7f71aaffd640 10 client.?.objecter ms_handle_connect 0x7f71b40a98f0 2024-03-25T17:38:20.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.898+0000 7f71aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:20.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767248894 0 0) 0x7f71b4138b20 con 0x7f71b40a98f0 2024-03-25T17:38:20.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:20.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:20.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:20.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 10 monclient(hunting): my global_id is 4647 2024-03-25T17:38:20.915 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.914+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7194004eb0 con 0x7f71b40a98f0 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.931+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2949907688 0 0) 0x7f7194004eb0 con 0x7f71b40a98f0 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b4138b20 con 0x7f71b40a98f0 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f71a4001bc0 con 0x7f71b40a98f0 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 10 monclient(hunting): dump: 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:20.933 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:20.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.932+0000 7f71aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3118769335 0 0) 0x7f71b4138b20 con 0x7f71b40a98f0 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 >> v1:172.21.15.96:6790/0 conn(0x7f71b413f710 legacy=0x7f71b4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 >> v1:172.21.15.203:6789/0 conn(0x7f71b41430c0 legacy=0x7f71b41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 1 monclient: found mon.a 2024-03-25T17:38:20.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4139a10 con 0x7f71b40a98f0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 10 monclient: _finish_auth 0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71b8cae640 5 monclient: authenticate success, global_id 4647 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71b8cae640 10 monclient: _renew_subs 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71b8cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.944+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b41dab00 con 0x7f71b40a98f0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 monclient: _renew_subs 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b413b4f0 con 0x7f71b40a98f0 2024-03-25T17:38:20.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 1 librados: init done 2024-03-25T17:38:20.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:20.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:20.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:20.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:20.945+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b40627b0 con 0x7f71b40a98f0 2024-03-25T17:38:21.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71a4004a90 con 0x7f71b40a98f0 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f71a4001250 con 0x7f71b40a98f0 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 10 monclient: dump: 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:21.261 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71b159a640 4 set_mon_vals no callback set 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.259+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f71a401ac90 con 0x7f71b40a98f0 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.260+0000 7f71abfff640 10 monclient: get_auth_request con 0x7f719406dcd0 auth_method 0 2024-03-25T17:38:21.262 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.260+0000 7f71aaffd640 10 client.4647.objecter ms_handle_connect 0x7f719406dcd0 2024-03-25T17:38:21.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f71a4054980 con 0x7f71b40a98f0 2024-03-25T17:38:21.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 10 client.4647.objecter ms_dispatch 0x7f71b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:21.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:21.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:21.266 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T17:38:21.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71a40502c0 con 0x7f71b40a98f0 2024-03-25T17:38:21.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:21.267 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.265+0000 7f71aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:21.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.522+0000 7f71b8cae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:21.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.522+0000 7f71b8cae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:21.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.522+0000 7f71b8cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:21.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.522+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71b40b3e80 con 0x7f71b40a98f0 2024-03-25T17:38:21.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.566+0000 7f71aaffd640 1 -- 172.21.15.96:0/3362650516 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f71a40523c0 con 0x7f71b40a98f0 2024-03-25T17:38:21.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.566+0000 7f71aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:21.567 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.566+0000 7f71aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:21.567 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:21.568 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:21.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 10 librados: watch_flush enter 2024-03-25T17:38:21.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 10 librados: watch_flush exit 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 10 client.4647.objecter successfully canceled tick 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 >> v1:172.21.15.203:6800/3611964061 conn(0x7f719406dcd0 legacy=0x7f7194070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 10 monclient: shutdown 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 >> v1:172.21.15.96:6789/0 conn(0x7f71b40a98f0 legacy=0x7f71b4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 shutdown_connections 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 >> 172.21.15.96:0/3362650516 conn(0x7f71b40b7ea0 msgr2=0x7f71b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 shutdown_connections 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 -- 172.21.15.96:0/3362650516 wait complete. 2024-03-25T17:38:21.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:21.568+0000 7f71b8cae640 1 librados: shutdown 2024-03-25T17:38:21.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:26.825 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:51.051 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.836 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:51.063 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.877 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:51.092 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.879 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:09:51.106 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.892 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:51.107 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.892 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:51.119 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.899 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:51.126 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.918 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:51.134 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:26.929 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:09:51.155 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:28.389 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:28.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.503+0000 7ffb19f0d640 10 monclient: get_monmap_and_config 2024-03-25T17:38:28.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.503+0000 7ffb19f0d640 10 monclient: build_initial_monmap 2024-03-25T17:38:28.504 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.503+0000 7ffb19f0d640 10 monclient: monmap: 2024-03-25T17:38:28.504 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:28.504 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:28.505129+0000 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:28.505129+0000 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.505 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.504+0000 7ffb19f0d640 1 Processor -- start 2024-03-25T17:38:28.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.504+0000 7ffb19f0d640 1 -- start start 2024-03-25T17:38:28.506 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.504+0000 7ffb19f0d640 10 monclient: init 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb140a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb141430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb14142520 con 0x7ffb140a98f0 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140b28a0 con 0x7ffb1413f710 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140b2a30 con 0x7ffb141430c0 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:28.507 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.505+0000 7ffb19f0d640 10 monclient(hunting): authenticate will time out at 5866.787758s 2024-03-25T17:38:28.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.519+0000 7ffb12ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ffb1413f710 0x7ffb141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:44014/0 (socket says 172.21.15.96:44014) 2024-03-25T17:38:28.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.519+0000 7ffb12ffd640 1 -- 172.21.15.96:0/400720162 learned_addr learned my addr 172.21.15.96:0/400720162 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:28.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.529+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124797669 0 0) 0x7ffb140b28a0 con 0x7ffb1413f710 2024-03-25T17:38:28.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.529+0000 7ffb127fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.529+0000 7ffb127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.530 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.529+0000 7ffb127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.529+0000 7ffb127fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T17:38:28.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.530+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc003520 con 0x7ffb1413f710 2024-03-25T17:38:28.538 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873281011 0 0) 0x7ffb140b2a30 con 0x7ffb141430c0 2024-03-25T17:38:28.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T17:38:28.539 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.538+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb140b28a0 con 0x7ffb141430c0 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381068349 0 0) 0x7ffb14142520 con 0x7ffb140a98f0 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T17:38:28.540 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.539+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb140b2a30 con 0x7ffb140a98f0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.558+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3370945308 0 0) 0x7ffb140b2a30 con 0x7ffb140a98f0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.558+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 >> v1:172.21.15.96:6790/0 conn(0x7ffb141430c0 legacy=0x7ffb1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.558+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 >> v1:172.21.15.203:6789/0 conn(0x7ffb1413f710 legacy=0x7ffb141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.558+0000 7ffb127fc640 10 monclient: _finish_hunting 0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 1 monclient: found mon.noname-a 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140b44c0 con 0x7ffb140a98f0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: _finish_auth 0 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: _check_auth_tickets 2024-03-25T17:38:28.560 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffb040015b0 con 0x7ffb140a98f0 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb19f0d640 5 monclient: authenticate success, global_id 4650 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb19f0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb127fc640 10 monclient: dump: 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:28.561 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:28.562 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:28.562 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.562 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.559+0000 7ffb19f0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:28.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb04001710 con 0x7ffb140a98f0 2024-03-25T17:38:28.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:28.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 1 -- 172.21.15.96:0/400720162 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffb04001b60 con 0x7ffb140a98f0 2024-03-25T17:38:28.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:28.588 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 4 set_mon_vals no callback set 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb127fc640 10 monclient: dump: 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:28.589 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 10 monclient: shutdown 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 1 -- 172.21.15.96:0/400720162 >> v1:172.21.15.96:6789/0 conn(0x7ffb140a98f0 legacy=0x7ffb140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 1 -- 172.21.15.96:0/400720162 shutdown_connections 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.588+0000 7ffb19f0d640 1 -- 172.21.15.96:0/400720162 >> 172.21.15.96:0/400720162 conn(0x7ffb140b7ea0 msgr2=0x7ffb140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.589+0000 7ffb19f0d640 1 -- 172.21.15.96:0/400720162 shutdown_connections 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.589+0000 7ffb19f0d640 1 -- 172.21.15.96:0/400720162 wait complete. 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 10 monclient: build_initial_monmap 2024-03-25T17:38:28.591 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 10 monclient: monmap: 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 librados: starting msgr at 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 librados: starting objecter 2024-03-25T17:38:28.592 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 Processor -- start 2024-03-25T17:38:28.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 -- start start 2024-03-25T17:38:28.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 librados: setting wanted keys 2024-03-25T17:38:28.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 1 librados: calling monclient init 2024-03-25T17:38:28.593 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.590+0000 7ffb19f0d640 10 monclient: init 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb140a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb141430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb141db270 con 0x7ffb141430c0 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb141db790 con 0x7ffb1413f710 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140af090 con 0x7ffb140a98f0 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:28.594 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.591+0000 7ffb19f0d640 10 monclient(hunting): authenticate will time out at 5866.873423s 2024-03-25T17:38:28.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.605+0000 7ffb137fe640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7ffb140a98f0 0x7ffb141da990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:41574/0 (socket says 172.21.15.96:41574) 2024-03-25T17:38:28.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.605+0000 7ffb137fe640 1 -- 172.21.15.96:0/2781392312 learned_addr learned my addr 172.21.15.96:0/2781392312 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:28.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.608+0000 7ffafbfff640 10 client.?.objecter ms_handle_connect 0x7ffb140a98f0 2024-03-25T17:38:28.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.608+0000 7ffafbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:28.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.615+0000 7ffafbfff640 10 client.?.objecter ms_handle_connect 0x7ffb141430c0 2024-03-25T17:38:28.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.615+0000 7ffafbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:28.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.616+0000 7ffafbfff640 10 client.?.objecter ms_handle_connect 0x7ffb1413f710 2024-03-25T17:38:28.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.616+0000 7ffafbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:28.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164710754 0 0) 0x7ffb141db790 con 0x7ffb1413f710 2024-03-25T17:38:28.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 10 monclient(hunting): my global_id is 4633 2024-03-25T17:38:28.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.625+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00004eb0 con 0x7ffb1413f710 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577464528 0 0) 0x7ffb140af090 con 0x7ffb140a98f0 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 10 monclient(hunting): my global_id is 4637 2024-03-25T17:38:28.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.627+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb141db790 con 0x7ffb140a98f0 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439773184 0 0) 0x7ffb141db790 con 0x7ffb140a98f0 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb140af090 con 0x7ffb140a98f0 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffb04001aa0 con 0x7ffb140a98f0 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 10 monclient(hunting): dump: 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.636 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.635+0000 7ffafbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356858634 0 0) 0x7ffb141db270 con 0x7ffb141430c0 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 10 monclient(hunting): my global_id is 4653 2024-03-25T17:38:28.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.638+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb141db790 con 0x7ffb141430c0 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2538277466 0 0) 0x7ffb00004eb0 con 0x7ffb1413f710 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb141db270 con 0x7ffb1413f710 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffb08001950 con 0x7ffb1413f710 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.645+0000 7ffafbfff640 10 monclient(hunting): dump: 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.646 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:28.647 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4017066236 0 0) 0x7ffb140af090 con 0x7ffb140a98f0 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 >> v1:172.21.15.203:6789/0 conn(0x7ffb1413f710 legacy=0x7ffb140ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 >> v1:172.21.15.96:6789/0 conn(0x7ffb141430c0 legacy=0x7ffb1414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 1 monclient: found mon.c 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:28.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140af220 con 0x7ffb140a98f0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 10 monclient: _finish_auth 0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffafbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 5 monclient: authenticate success, global_id 4637 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 10 monclient: _renew_subs 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1414de80 con 0x7ffb140a98f0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 10 monclient: _renew_subs 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1414e3b0 con 0x7ffb140a98f0 2024-03-25T17:38:28.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.647+0000 7ffb19f0d640 1 librados: init done 2024-03-25T17:38:28.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.648+0000 7ffb19f0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:28.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.648+0000 7ffb19f0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:28.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.648+0000 7ffb19f0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:38:28.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.648+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb141db400 con 0x7ffb140a98f0 2024-03-25T17:38:28.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb040038a0 con 0x7ffb140a98f0 2024-03-25T17:38:28.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:28.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7ffb04003cf0 con 0x7ffb140a98f0 2024-03-25T17:38:28.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffafbfff640 10 monclient: dump: 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:28.673 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.671+0000 7ffb11ffb640 4 set_mon_vals no callback set 2024-03-25T17:38:28.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.672+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7ffb04003f10 con 0x7ffb140a98f0 2024-03-25T17:38:28.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.673+0000 7ffb12ffd640 10 monclient: get_auth_request con 0x7ffb0006dfb0 auth_method 0 2024-03-25T17:38:28.674 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.673+0000 7ffafbfff640 10 client.4637.objecter ms_handle_connect 0x7ffb0006dfb0 2024-03-25T17:38:28.676 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.676+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7ffb04054840 con 0x7ffb140a98f0 2024-03-25T17:38:28.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.676+0000 7ffafbfff640 10 client.4637.objecter ms_dispatch 0x7ffb14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:28.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.676+0000 7ffafbfff640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:28.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.676+0000 7ffafbfff640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:28.677 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.676+0000 7ffafbfff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T17:38:28.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.971+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb04050350 con 0x7ffb140a98f0 2024-03-25T17:38:28.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.971+0000 7ffafbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:28.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:28.971+0000 7ffafbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:29.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.202+0000 7ffb19f0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:29.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.202+0000 7ffb19f0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:29.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.202+0000 7ffb19f0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:38:29.202 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.202+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb140b0e70 con 0x7ffb140a98f0 2024-03-25T17:38:29.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.231+0000 7ffafbfff640 1 -- 172.21.15.96:0/2781392312 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7ffb04052280 con 0x7ffb140a98f0 2024-03-25T17:38:29.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.231+0000 7ffafbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:29.231 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.231+0000 7ffafbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:29.231 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:29.231 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 10 librados: watch_flush enter 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 10 librados: watch_flush exit 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 10 client.4637.objecter successfully canceled tick 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 >> v1:172.21.15.203:6800/3611964061 conn(0x7ffb0006dfb0 legacy=0x7ffb00070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 10 monclient: shutdown 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:29.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 >> v1:172.21.15.96:6790/0 conn(0x7ffb140a98f0 legacy=0x7ffb141da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:29.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 shutdown_connections 2024-03-25T17:38:29.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 >> 172.21.15.96:0/2781392312 conn(0x7ffb140b7ea0 msgr2=0x7ffb14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:29.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 shutdown_connections 2024-03-25T17:38:29.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 -- 172.21.15.96:0/2781392312 wait complete. 2024-03-25T17:38:29.235 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:29.233+0000 7ffb19f0d640 1 librados: shutdown 2024-03-25T17:38:29.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:36.052 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:36.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.168+0000 7f39c8d77640 10 monclient: get_monmap_and_config 2024-03-25T17:38:36.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.168+0000 7f39c8d77640 10 monclient: build_initial_monmap 2024-03-25T17:38:36.168 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.168+0000 7f39c8d77640 10 monclient: monmap: 2024-03-25T17:38:36.168 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:36.168 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:36.168841+0000 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:36.168841+0000 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:36.169 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.169+0000 7f39c8d77640 1 Processor -- start 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.169+0000 7f39c8d77640 1 -- start start 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.169+0000 7f39c8d77640 10 monclient: init 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-b con 0x7f39c40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:36.170 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-c con 0x7f39c413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-a con 0x7f39c41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c4142520 con 0x7f39c41430c0 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40b28a0 con 0x7f39c413f710 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40b2a30 con 0x7f39c40a98f0 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.170+0000 7f39c8d77640 10 monclient(hunting): authenticate will time out at 5874.451357s 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.171+0000 7f39c259c640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f39c40a98f0 0x7f39c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:55772/0 (socket says 172.21.15.96:55772) 2024-03-25T17:38:36.171 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.171+0000 7f39c259c640 1 -- 172.21.15.96:0/3797442678 learned_addr learned my addr 172.21.15.96:0/3797442678 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058795192 0 0) 0x7f39c4142520 con 0x7f39c41430c0 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 10 monclient(hunting): _negotiate 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 10 monclient(hunting): my global_id is 4656 2024-03-25T17:38:36.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.190+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac003520 con 0x7f39c41430c0 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614383359 0 0) 0x7f39c40b2a30 con 0x7f39c40a98f0 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 10 monclient(hunting): _negotiate 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 10 monclient(hunting): my global_id is 4640 2024-03-25T17:38:36.191 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.191+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c4142520 con 0x7f39c40a98f0 2024-03-25T17:38:36.209 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183255063 0 0) 0x7f39ac003520 con 0x7f39c41430c0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 >> v1:172.21.15.96:6790/0 conn(0x7f39c40a98f0 legacy=0x7f39c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 >> v1:172.21.15.203:6789/0 conn(0x7f39c413f710 legacy=0x7f39c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: _finish_hunting 0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 monclient: found mon.noname-a 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40b44c0 con 0x7f39c41430c0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: _finish_auth 0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: _check_auth_tickets 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f39bc001a10 con 0x7f39c41430c0 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:36.210 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c159a640 10 monclient: dump: 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c8d77640 5 monclient: authenticate success, global_id 4656 2024-03-25T17:38:36.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.209+0000 7f39c8d77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:36.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.222+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39bc001b70 con 0x7f39c41430c0 2024-03-25T17:38:36.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.222+0000 7f39c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.222+0000 7f39c159a640 1 -- 172.21.15.96:0/3797442678 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f39bc001f80 con 0x7f39c41430c0 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.222+0000 7f39c8d77640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.222+0000 7f39c8d77640 4 set_mon_vals no callback set 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c159a640 10 monclient: dump: 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:36.223 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:36.224 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 10 monclient: shutdown 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 1 -- 172.21.15.96:0/3797442678 >> v1:172.21.15.96:6789/0 conn(0x7f39c41430c0 legacy=0x7f39c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 1 -- 172.21.15.96:0/3797442678 shutdown_connections 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 1 -- 172.21.15.96:0/3797442678 >> 172.21.15.96:0/3797442678 conn(0x7f39c40b7ea0 msgr2=0x7f39c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 1 -- 172.21.15.96:0/3797442678 shutdown_connections 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.223+0000 7f39c8d77640 1 -- 172.21.15.96:0/3797442678 wait complete. 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.224+0000 7f39c8d77640 10 monclient: build_initial_monmap 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.224+0000 7f39c8d77640 10 monclient: monmap: 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:36.226 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 librados: starting msgr at 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 librados: starting objecter 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 Processor -- start 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 -- start start 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 librados: setting wanted keys 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 librados: calling monclient init 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient: init 2024-03-25T17:38:36.227 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-a con 0x7f39c40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-c con 0x7f39c413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient(hunting): picked mon.noname-b con 0x7f39c41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c4136a50 con 0x7f39c40a98f0 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40b79c0 con 0x7f39c41430c0 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40b7ca0 con 0x7f39c413f710 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:36.228 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.225+0000 7f39c8d77640 10 monclient(hunting): authenticate will time out at 5874.506744s 2024-03-25T17:38:36.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.244+0000 7f39c259c640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f39c40a98f0 0x7f39c4137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55174/0 (socket says 172.21.15.96:55174) 2024-03-25T17:38:36.243 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.244+0000 7f39c259c640 1 -- 172.21.15.96:0/4065787820 learned_addr learned my addr 172.21.15.96:0/4065787820 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:36.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.246+0000 7f39b2ffd640 10 client.?.objecter ms_handle_connect 0x7f39c41430c0 2024-03-25T17:38:36.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.246+0000 7f39b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:36.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.247+0000 7f39b2ffd640 10 client.?.objecter ms_handle_connect 0x7f39c40a98f0 2024-03-25T17:38:36.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.247+0000 7f39b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:36.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.257+0000 7f39b2ffd640 10 client.?.objecter ms_handle_connect 0x7f39c413f710 2024-03-25T17:38:36.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.257+0000 7f39b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792837243 0 0) 0x7f39c4136a50 con 0x7f39c40a98f0 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 10 monclient(hunting): my global_id is 4659 2024-03-25T17:38:36.261 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.261+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a4004eb0 con 0x7f39c40a98f0 2024-03-25T17:38:36.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680158019 0 0) 0x7f39c40b79c0 con 0x7f39c41430c0 2024-03-25T17:38:36.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:36.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:36.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:36.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 10 monclient(hunting): my global_id is 4639 2024-03-25T17:38:36.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.270+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c4136a50 con 0x7f39c41430c0 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519497946 0 0) 0x7f39a4004eb0 con 0x7f39c40a98f0 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c40b79c0 con 0x7f39c40a98f0 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f39b40019b0 con 0x7f39c40a98f0 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 10 monclient(hunting): dump: 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:36.272 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:36.273 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:36.273 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:36.273 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.273 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.272+0000 7f39b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:36.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.280+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092342010 0 0) 0x7f39c40b7ca0 con 0x7f39c413f710 2024-03-25T17:38:36.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.280+0000 7f39b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:36.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.280+0000 7f39b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:36.280 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.280+0000 7f39b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:36.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.280+0000 7f39b2ffd640 10 monclient(hunting): my global_id is 4643 2024-03-25T17:38:36.281 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.281+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a4004eb0 con 0x7f39c413f710 2024-03-25T17:38:36.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4281406590 0 0) 0x7f39c40b79c0 con 0x7f39c40a98f0 2024-03-25T17:38:36.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 >> v1:172.21.15.96:6790/0 conn(0x7f39c413f710 legacy=0x7f39c4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 >> v1:172.21.15.203:6789/0 conn(0x7f39c41430c0 legacy=0x7f39c41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 1 monclient: found mon.a 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.287+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4136ca0 con 0x7f39c40a98f0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39b2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 5 monclient: authenticate success, global_id 4659 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 10 monclient: _renew_subs 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c4136f80 con 0x7f39c40a98f0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 10 monclient: _renew_subs 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c41360e0 con 0x7f39c40a98f0 2024-03-25T17:38:36.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.288+0000 7f39c8d77640 1 librados: init done 2024-03-25T17:38:36.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.289+0000 7f39c8d77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:36.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.289+0000 7f39c8d77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:36.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.289+0000 7f39c8d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.289+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c40627b0 con 0x7f39c40a98f0 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39b40047e0 con 0x7f39c40a98f0 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f39b4004b30 con 0x7f39c40a98f0 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39b2ffd640 10 monclient: dump: 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:36.599 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:36.600 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.599+0000 7f39c0d99640 4 set_mon_vals no callback set 2024-03-25T17:38:36.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.600+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f39b4004d50 con 0x7f39c40a98f0 2024-03-25T17:38:36.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.601+0000 7f39c1d9b640 10 monclient: get_auth_request con 0x7f39a406de00 auth_method 0 2024-03-25T17:38:36.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.601+0000 7f39b2ffd640 10 client.4659.objecter ms_handle_connect 0x7f39a406de00 2024-03-25T17:38:36.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f39b4054890 con 0x7f39c40a98f0 2024-03-25T17:38:36.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 10 client.4659.objecter ms_dispatch 0x7f39c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39b40503a0 con 0x7f39c40a98f0 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:36.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.608+0000 7f39b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:36.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.869+0000 7f39c8d77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:36.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.869+0000 7f39c8d77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:36.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.869+0000 7f39c8d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:38:36.869 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.869+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39c413e290 con 0x7f39c40a98f0 2024-03-25T17:38:36.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.944+0000 7f39b2ffd640 1 -- 172.21.15.96:0/4065787820 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f39b40522d0 con 0x7f39c40a98f0 2024-03-25T17:38:36.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.944+0000 7f39b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:36.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.944+0000 7f39b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:36.944 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:36.945 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:36.946 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 10 librados: watch_flush enter 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 10 librados: watch_flush exit 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 10 client.4659.objecter successfully canceled tick 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 >> v1:172.21.15.203:6800/3611964061 conn(0x7f39a406de00 legacy=0x7f39a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 10 monclient: shutdown 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 >> v1:172.21.15.96:6789/0 conn(0x7f39c40a98f0 legacy=0x7f39c4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 shutdown_connections 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.946+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 >> 172.21.15.96:0/4065787820 conn(0x7f39c40b7ea0 msgr2=0x7f39c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.947+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 shutdown_connections 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.947+0000 7f39c8d77640 1 -- 172.21.15.96:0/4065787820 wait complete. 2024-03-25T17:38:36.947 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:36.947+0000 7f39c8d77640 1 librados: shutdown 2024-03-25T17:38:36.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:43.764 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:43.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.880+0000 7fae318c1640 10 monclient: get_monmap_and_config 2024-03-25T17:38:43.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.880+0000 7fae318c1640 10 monclient: build_initial_monmap 2024-03-25T17:38:43.906 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.880+0000 7fae318c1640 10 monclient: monmap: 2024-03-25T17:38:43.906 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:43.906 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:43.906 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:43.882149+0000 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:43.882149+0000 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:43.907 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:43.909 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.881+0000 7fae318c1640 1 Processor -- start 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.881+0000 7fae318c1640 1 -- start start 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.881+0000 7fae318c1640 10 monclient: init 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-b con 0x7fae2c0a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-c con 0x7fae2c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-a con 0x7fae2c1430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c142520 con 0x7fae2c1430c0 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c0b28a0 con 0x7fae2c13f710 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c0b2a30 con 0x7fae2c0a98f0 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:43.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.882+0000 7fae318c1640 10 monclient(hunting): authenticate will time out at 5882.164418s 2024-03-25T17:38:43.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.890+0000 7fae2a7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fae2c13f710 0x7fae2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:42224/0 (socket says 172.21.15.96:42224) 2024-03-25T17:38:43.912 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.890+0000 7fae2a7fc640 1 -- 172.21.15.96:0/520190014 learned_addr learned my addr 172.21.15.96:0/520190014 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248204811 0 0) 0x7fae2c0b28a0 con 0x7fae2c13f710 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 10 monclient(hunting): my global_id is 4642 2024-03-25T17:38:43.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.913+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae14003520 con 0x7fae2c13f710 2024-03-25T17:38:43.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880119404 0 0) 0x7fae2c0b2a30 con 0x7fae2c0a98f0 2024-03-25T17:38:43.921 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:43.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:43.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:43.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 10 monclient(hunting): my global_id is 4646 2024-03-25T17:38:43.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.921+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c0b28a0 con 0x7fae2c0a98f0 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.925+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621477455 0 0) 0x7fae14003520 con 0x7fae2c13f710 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.925+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 >> v1:172.21.15.96:6790/0 conn(0x7fae2c0a98f0 legacy=0x7fae2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 >> v1:172.21.15.96:6789/0 conn(0x7fae2c1430c0 legacy=0x7fae2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 1 monclient: found mon.noname-c 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c0b44c0 con 0x7fae2c13f710 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: _finish_auth 0 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:43.927 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae20001690 con 0x7fae2c13f710 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae29ffb640 10 monclient: dump: 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae318c1640 5 monclient: authenticate success, global_id 4642 2024-03-25T17:38:43.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.926+0000 7fae318c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:43.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae200017f0 con 0x7fae2c13f710 2024-03-25T17:38:43.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 1 -- 172.21.15.96:0/520190014 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae20001c00 con 0x7fae2c13f710 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae318c1640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae318c1640 4 set_mon_vals no callback set 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae29ffb640 10 monclient: dump: 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:43.937 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae318c1640 10 monclient: shutdown 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae318c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.936+0000 7fae318c1640 1 -- 172.21.15.96:0/520190014 >> v1:172.21.15.203:6789/0 conn(0x7fae2c13f710 legacy=0x7fae2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 1 -- 172.21.15.96:0/520190014 shutdown_connections 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 1 -- 172.21.15.96:0/520190014 >> 172.21.15.96:0/520190014 conn(0x7fae2c0b7ea0 msgr2=0x7fae2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 1 -- 172.21.15.96:0/520190014 shutdown_connections 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 1 -- 172.21.15.96:0/520190014 wait complete. 2024-03-25T17:38:43.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 10 monclient: build_initial_monmap 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.937+0000 7fae318c1640 10 monclient: monmap: 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 librados: starting msgr at 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 librados: starting objecter 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 Processor -- start 2024-03-25T17:38:43.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 -- start start 2024-03-25T17:38:43.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 librados: setting wanted keys 2024-03-25T17:38:43.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 1 librados: calling monclient init 2024-03-25T17:38:43.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.938+0000 7fae318c1640 10 monclient: init 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-a con 0x7fae2c0a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-c con 0x7fae2c13f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient(hunting): picked mon.noname-b con 0x7fae2c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c1d77a0 con 0x7fae2c0a98f0 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c138e40 con 0x7fae2c1430c0 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c0aba30 con 0x7fae2c13f710 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:43.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.939+0000 7fae318c1640 10 monclient(hunting): authenticate will time out at 5882.221473s 2024-03-25T17:38:43.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.950+0000 7fae2b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fae2c1430c0 0x7fae2c134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:42274/0 (socket says 172.21.15.96:42274) 2024-03-25T17:38:43.951 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.950+0000 7fae2b7fe640 1 -- 172.21.15.96:0/2857572623 learned_addr learned my addr 172.21.15.96:0/2857572623 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:43.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.956+0000 7fae13fff640 10 client.?.objecter ms_handle_connect 0x7fae2c1430c0 2024-03-25T17:38:43.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.956+0000 7fae13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:43.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.964+0000 7fae13fff640 10 client.?.objecter ms_handle_connect 0x7fae2c13f710 2024-03-25T17:38:43.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.964+0000 7fae13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:43.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.968+0000 7fae13fff640 10 client.?.objecter ms_handle_connect 0x7fae2c0a98f0 2024-03-25T17:38:43.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.968+0000 7fae13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:43.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.971+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383917384 0 0) 0x7fae2c138e40 con 0x7fae2c1430c0 2024-03-25T17:38:43.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.972+0000 7fae13fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:43.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.972+0000 7fae13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:43.972 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.972+0000 7fae13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:43.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.972+0000 7fae13fff640 10 monclient(hunting): my global_id is 4645 2024-03-25T17:38:43.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.972+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae18004eb0 con 0x7fae2c1430c0 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916799569 0 0) 0x7fae2c0aba30 con 0x7fae2c13f710 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 10 monclient(hunting): my global_id is 4649 2024-03-25T17:38:43.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.978+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c138e40 con 0x7fae2c13f710 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198741290 0 0) 0x7fae2c1d77a0 con 0x7fae2c0a98f0 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 10 monclient(hunting): my global_id is 4665 2024-03-25T17:38:43.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:43.980+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c0aba30 con 0x7fae2c0a98f0 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423343475 0 0) 0x7fae18004eb0 con 0x7fae2c1430c0 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae2c1d77a0 con 0x7fae2c1430c0 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae24001c30 con 0x7fae2c1430c0 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 10 monclient(hunting): dump: 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:44.008 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:44.009 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:44.009 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:44.009 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:44.009 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.007+0000 7fae13fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843247518 0 0) 0x7fae2c138e40 con 0x7fae2c13f710 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae18004eb0 con 0x7fae2c13f710 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae20001b40 con 0x7fae2c13f710 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.009+0000 7fae13fff640 10 monclient(hunting): dump: 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:44.010 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:44.011 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:44.011 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:44.011 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2927525899 0 0) 0x7fae2c0aba30 con 0x7fae2c0a98f0 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae2c138e40 con 0x7fae2c0a98f0 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae1c001930 con 0x7fae2c0a98f0 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.012+0000 7fae13fff640 10 monclient(hunting): dump: 2024-03-25T17:38:44.013 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:44.014 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1282529418 0 0) 0x7fae2c1d77a0 con 0x7fae2c1430c0 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 >> v1:172.21.15.96:6790/0 conn(0x7fae2c13f710 legacy=0x7fae2c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 >> v1:172.21.15.96:6789/0 conn(0x7fae2c0a98f0 legacy=0x7fae2c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 1 monclient: found mon.b 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c1d8340 con 0x7fae2c1430c0 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 10 monclient: _finish_auth 0 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:44.029 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 5 monclient: authenticate success, global_id 4645 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 10 monclient: _renew_subs 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c134820 con 0x7fae2c1430c0 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 10 monclient: _renew_subs 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c1380c0 con 0x7fae2c1430c0 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.028+0000 7fae318c1640 1 librados: init done 2024-03-25T17:38:44.030 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.029+0000 7fae318c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:44.031 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.029+0000 7fae318c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:44.031 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.029+0000 7fae318c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:44.031 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.029+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c0627b0 con 0x7fae2c1430c0 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae24003990 con 0x7fae2c1430c0 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fae24003da0 con 0x7fae2c1430c0 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae13fff640 10 monclient: dump: 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:44.352 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.351+0000 7fae297fa640 4 set_mon_vals no callback set 2024-03-25T17:38:44.353 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.352+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fae24003fc0 con 0x7fae2c1430c0 2024-03-25T17:38:44.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.353+0000 7fae2affd640 10 monclient: get_auth_request con 0x7fae1806e0f0 auth_method 0 2024-03-25T17:38:44.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.353+0000 7fae13fff640 10 client.4645.objecter ms_handle_connect 0x7fae1806e0f0 2024-03-25T17:38:44.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fae24054a30 con 0x7fae2c1430c0 2024-03-25T17:38:44.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 10 client.4645.objecter ms_dispatch 0x7fae2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae24050540 con 0x7fae2c1430c0 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:44.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.358+0000 7fae13fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:44.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.613+0000 7fae318c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:44.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.613+0000 7fae318c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:44.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.613+0000 7fae318c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:44.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.613+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae2c14e880 con 0x7fae2c1430c0 2024-03-25T17:38:44.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.643+0000 7fae13fff640 1 -- 172.21.15.96:0/2857572623 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fae24052470 con 0x7fae2c1430c0 2024-03-25T17:38:44.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.643+0000 7fae13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:44.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.643+0000 7fae13fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:44.644 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:44.645 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 10 librados: watch_flush enter 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 10 librados: watch_flush exit 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 10 client.4645.objecter successfully canceled tick 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 >> v1:172.21.15.203:6800/3611964061 conn(0x7fae1806e0f0 legacy=0x7fae180705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 10 monclient: shutdown 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.645+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 >> v1:172.21.15.203:6789/0 conn(0x7fae2c1430c0 legacy=0x7fae2c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.646+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 shutdown_connections 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.646+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 >> 172.21.15.96:0/2857572623 conn(0x7fae2c0b7ea0 msgr2=0x7fae2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:44.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.646+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 shutdown_connections 2024-03-25T17:38:44.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.646+0000 7fae318c1640 1 -- 172.21.15.96:0/2857572623 wait complete. 2024-03-25T17:38:44.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:44.646+0000 7fae318c1640 1 librados: shutdown 2024-03-25T17:38:44.657 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:46.872 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:11.099 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.915 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:11.131 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.916 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:11.132 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.926 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:11.153 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.927 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:11.142 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.971 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:11.197 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.973 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:11.200 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:46.989 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:11.216 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:47.016 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:11.243 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:51.466 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:51.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.581+0000 7fc359a25640 10 monclient: get_monmap_and_config 2024-03-25T17:38:51.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.581+0000 7fc359a25640 10 monclient: build_initial_monmap 2024-03-25T17:38:51.582 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.581+0000 7fc359a25640 10 monclient: monmap: 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:51.582944+0000 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:51.582944+0000 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:51.583 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.582+0000 7fc359a25640 1 Processor -- start 2024-03-25T17:38:51.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.582+0000 7fc359a25640 1 -- start start 2024-03-25T17:38:51.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.582+0000 7fc359a25640 10 monclient: init 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-b con 0x7fc35406c380 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-c con 0x7fc3540b12d0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-a con 0x7fc3540aedf0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3540b16f0 con 0x7fc3540aedf0 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3540adfe0 con 0x7fc3540b12d0 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3540add00 con 0x7fc35406c380 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:51.586 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.583+0000 7fc359a25640 10 monclient(hunting): authenticate will time out at 5889.865661s 2024-03-25T17:38:51.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.588+0000 7fc3527fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc3540b12d0 0x7fc3540ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:48836/0 (socket says 172.21.15.96:48836) 2024-03-25T17:38:51.589 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.588+0000 7fc3527fc640 1 -- 172.21.15.96:0/4075564661 learned_addr learned my addr 172.21.15.96:0/4075564661 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:51.595 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.594+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753467423 0 0) 0x7fc3540adfe0 con 0x7fc3540b12d0 2024-03-25T17:38:51.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.595+0000 7fc351ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:51.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.595+0000 7fc351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:51.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.595+0000 7fc351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:51.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.595+0000 7fc351ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T17:38:51.596 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.595+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c003520 con 0x7fc3540b12d0 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577689229 0 0) 0x7fc3540b16f0 con 0x7fc3540aedf0 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 10 monclient(hunting): my global_id is 4668 2024-03-25T17:38:51.598 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.597+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3540adfe0 con 0x7fc3540aedf0 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163408957 0 0) 0x7fc3540adfe0 con 0x7fc3540aedf0 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 >> v1:172.21.15.96:6790/0 conn(0x7fc35406c380 legacy=0x7fc3540b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 >> v1:172.21.15.203:6789/0 conn(0x7fc3540b12d0 legacy=0x7fc3540ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 monclient: found mon.noname-a 2024-03-25T17:38:51.605 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3540ae2c0 con 0x7fc3540aedf0 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: _finish_auth 0 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc34c0019f0 con 0x7fc3540aedf0 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc359a25640 5 monclient: authenticate success, global_id 4668 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc359a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:51.606 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc351ffb640 10 monclient: dump: 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:51.607 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:51.608 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.608 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.604+0000 7fc359a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:51.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.612+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc34c001b50 con 0x7fc3540aedf0 2024-03-25T17:38:51.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc351ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:51.613 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc351ffb640 1 -- 172.21.15.96:0/4075564661 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc34c001f20 con 0x7fc3540aedf0 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc359a25640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc359a25640 4 set_mon_vals no callback set 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc351ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc351ffb640 10 monclient: dump: 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.614 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:51.615 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:51.615 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:51.615 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:51.615 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:51.615 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc359a25640 10 monclient: shutdown 2024-03-25T17:38:51.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc359a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:51.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.613+0000 7fc359a25640 1 -- 172.21.15.96:0/4075564661 >> v1:172.21.15.96:6789/0 conn(0x7fc3540aedf0 legacy=0x7fc35414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.614+0000 7fc359a25640 1 -- 172.21.15.96:0/4075564661 shutdown_connections 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.614+0000 7fc359a25640 1 -- 172.21.15.96:0/4075564661 >> 172.21.15.96:0/4075564661 conn(0x7fc3540b5d10 msgr2=0x7fc3540b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.614+0000 7fc359a25640 1 -- 172.21.15.96:0/4075564661 shutdown_connections 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.614+0000 7fc359a25640 1 -- 172.21.15.96:0/4075564661 wait complete. 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 10 monclient: build_initial_monmap 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 10 monclient: monmap: 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:51.618 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 librados: starting msgr at 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 librados: starting objecter 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 Processor -- start 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 -- start start 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 librados: setting wanted keys 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 1 librados: calling monclient init 2024-03-25T17:38:51.619 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.615+0000 7fc359a25640 10 monclient: init 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-b con 0x7fc35406c380 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-a con 0x7fc3540aedf0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient(hunting): picked mon.noname-c con 0x7fc3540b12d0 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35406c8e0 con 0x7fc3540aedf0 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3541dae80 con 0x7fc35406c380 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3541c0ef0 con 0x7fc3540b12d0 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:51.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.616+0000 7fc359a25640 10 monclient(hunting): authenticate will time out at 5889.898691s 2024-03-25T17:38:51.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.624+0000 7fc352ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc35406c380 0x7fc3541da770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:48852/0 (socket says 172.21.15.96:48852) 2024-03-25T17:38:51.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.624+0000 7fc352ffd640 1 -- 172.21.15.96:0/3724450699 learned_addr learned my addr 172.21.15.96:0/3724450699 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:51.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.626+0000 7fc33bfff640 10 client.?.objecter ms_handle_connect 0x7fc35406c380 2024-03-25T17:38:51.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.626+0000 7fc33bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:51.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.628+0000 7fc33bfff640 10 client.?.objecter ms_handle_connect 0x7fc3540aedf0 2024-03-25T17:38:51.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.628+0000 7fc33bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:51.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.633+0000 7fc33bfff640 10 client.?.objecter ms_handle_connect 0x7fc3540b12d0 2024-03-25T17:38:51.633 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.633+0000 7fc33bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340323487 0 0) 0x7fc3541dae80 con 0x7fc35406c380 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 10 monclient(hunting): my global_id is 4651 2024-03-25T17:38:51.642 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.641+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340004eb0 con 0x7fc35406c380 2024-03-25T17:38:51.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71560178 0 0) 0x7fc35406c8e0 con 0x7fc3540aedf0 2024-03-25T17:38:51.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:51.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:51.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:51.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 10 monclient(hunting): my global_id is 4671 2024-03-25T17:38:51.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.649+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3541dae80 con 0x7fc3540aedf0 2024-03-25T17:38:51.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494484220 0 0) 0x7fc3541c0ef0 con 0x7fc3540b12d0 2024-03-25T17:38:51.660 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:51.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:51.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:51.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 10 monclient(hunting): my global_id is 4655 2024-03-25T17:38:51.661 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.659+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35406c8e0 con 0x7fc3540b12d0 2024-03-25T17:38:51.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352083068 0 0) 0x7fc340004eb0 con 0x7fc35406c380 2024-03-25T17:38:51.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3541c0ef0 con 0x7fc35406c380 2024-03-25T17:38:51.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc344001a30 con 0x7fc35406c380 2024-03-25T17:38:51.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:51.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 10 monclient(hunting): dump: 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.666 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.664+0000 7fc33bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:51.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 755330962 0 0) 0x7fc3541c0ef0 con 0x7fc35406c380 2024-03-25T17:38:51.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 >> v1:172.21.15.96:6790/0 conn(0x7fc3540b12d0 legacy=0x7fc3541c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:51.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 >> v1:172.21.15.96:6789/0 conn(0x7fc3540aedf0 legacy=0x7fc3541c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:51.688 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 1 monclient: found mon.b 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3541c1080 con 0x7fc35406c380 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 10 monclient: _finish_auth 0 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc33bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 5 monclient: authenticate success, global_id 4651 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 10 monclient: _renew_subs 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3541c1850 con 0x7fc35406c380 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:51.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 10 monclient: _renew_subs 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3541c2900 con 0x7fc35406c380 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.686+0000 7fc359a25640 1 librados: init done 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.687+0000 7fc359a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.687+0000 7fc359a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.687+0000 7fc359a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:51.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.687+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3541c2c00 con 0x7fc35406c380 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc344001c70 con 0x7fc35406c380 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fc344003a60 con 0x7fc35406c380 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 10 monclient: dump: 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:51.982 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:51.983 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc3517fa640 4 set_mon_vals no callback set 2024-03-25T17:38:51.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.981+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fc344003c40 con 0x7fc35406c380 2024-03-25T17:38:51.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.982+0000 7fc3527fc640 10 monclient: get_auth_request con 0x7fc34006de20 auth_method 0 2024-03-25T17:38:51.984 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.982+0000 7fc33bfff640 10 client.4651.objecter ms_handle_connect 0x7fc34006de20 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7fc344054900 con 0x7fc35406c380 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 10 client.4651.objecter ms_dispatch 0x7fc354000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc344050240 con 0x7fc35406c380 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:51.987 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:51.986+0000 7fc33bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:52.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.216+0000 7fc359a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:52.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.216+0000 7fc359a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:52.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.216+0000 7fc359a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:52.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.216+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3541db010 con 0x7fc35406c380 2024-03-25T17:38:52.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.267+0000 7fc33bfff640 1 -- 172.21.15.96:0/3724450699 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7fc344052340 con 0x7fc35406c380 2024-03-25T17:38:52.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.267+0000 7fc33bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:52.268 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.267+0000 7fc33bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:52.269 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:52.269 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 10 librados: watch_flush enter 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 10 librados: watch_flush exit 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 10 client.4651.objecter successfully canceled tick 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 >> v1:172.21.15.203:6800/3611964061 conn(0x7fc34006de20 legacy=0x7fc3400702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:52.271 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 10 monclient: shutdown 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 >> v1:172.21.15.203:6789/0 conn(0x7fc35406c380 legacy=0x7fc3541da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 shutdown_connections 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 >> 172.21.15.96:0/3724450699 conn(0x7fc3540b5d10 msgr2=0x7fc3540b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 shutdown_connections 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 -- 172.21.15.96:0/3724450699 wait complete. 2024-03-25T17:38:52.272 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:52.270+0000 7fc359a25640 1 librados: shutdown 2024-03-25T17:38:52.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:59.089 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:59.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.202+0000 7f473904a640 10 monclient: get_monmap_and_config 2024-03-25T17:38:59.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.202+0000 7f473904a640 10 monclient: build_initial_monmap 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.202+0000 7f473904a640 10 monclient: monmap: 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:38:59.204142+0000 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:38:59.204142+0000 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:59.204 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:38:59.205 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:38:59.205 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.203+0000 7f473904a640 1 Processor -- start 2024-03-25T17:38:59.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.203+0000 7f473904a640 1 -- start start 2024-03-25T17:38:59.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.203+0000 7f473904a640 10 monclient: init 2024-03-25T17:38:59.206 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient(hunting): picked mon.noname-a con 0x7f47340a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient(hunting): picked mon.noname-b con 0x7f473413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient(hunting): picked mon.noname-c con 0x7f47341430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4734142520 con 0x7f47340a98f0 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340b28a0 con 0x7f47341430c0 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340b2a30 con 0x7f473413f710 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:59.207 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.204+0000 7f473904a640 10 monclient(hunting): authenticate will time out at 5897.486215s 2024-03-25T17:38:59.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.215+0000 7f4732d9d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f47340a98f0 0x7f47340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48234/0 (socket says 172.21.15.96:48234) 2024-03-25T17:38:59.216 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.215+0000 7f4732d9d640 1 -- 172.21.15.96:0/1593989841 learned_addr learned my addr 172.21.15.96:0/1593989841 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844570854 0 0) 0x7f47340b2a30 con 0x7f473413f710 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 10 monclient(hunting): my global_id is 4658 2024-03-25T17:38:59.234 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.233+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c003520 con 0x7f473413f710 2024-03-25T17:38:59.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975466709 0 0) 0x7f4734142520 con 0x7f47340a98f0 2024-03-25T17:38:59.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.236 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 10 monclient(hunting): my global_id is 4674 2024-03-25T17:38:59.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.235+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47340b2a30 con 0x7f47340a98f0 2024-03-25T17:38:59.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858496754 0 0) 0x7f47340b28a0 con 0x7f47341430c0 2024-03-25T17:38:59.237 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 10 monclient(hunting): my global_id is 4654 2024-03-25T17:38:59.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.236+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734142520 con 0x7f47341430c0 2024-03-25T17:38:59.257 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3802039564 0 0) 0x7f471c003520 con 0x7f473413f710 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 >> v1:172.21.15.203:6789/0 conn(0x7f47341430c0 legacy=0x7f473414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 >> v1:172.21.15.96:6789/0 conn(0x7f47340a98f0 legacy=0x7f47340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 monclient: found mon.noname-b 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340b44c0 con 0x7f473413f710 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: _finish_auth 0 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:38:59.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4728001690 con 0x7f473413f710 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f4731d9b640 10 monclient: dump: 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:59.259 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:59.260 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:59.260 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:59.260 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f473904a640 5 monclient: authenticate success, global_id 4658 2024-03-25T17:38:59.260 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.256+0000 7f473904a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:59.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47280017f0 con 0x7f473413f710 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 1 -- 172.21.15.96:0/1593989841 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4728001c00 con 0x7f473413f710 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f473904a640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f473904a640 4 set_mon_vals no callback set 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.269+0000 7f4731d9b640 10 monclient: dump: 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:59.270 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:59.271 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 10 monclient: shutdown 2024-03-25T17:38:59.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:59.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 1 -- 172.21.15.96:0/1593989841 >> v1:172.21.15.96:6790/0 conn(0x7f473413f710 legacy=0x7f47341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 1 -- 172.21.15.96:0/1593989841 shutdown_connections 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 1 -- 172.21.15.96:0/1593989841 >> 172.21.15.96:0/1593989841 conn(0x7f47340b7ea0 msgr2=0x7f47340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 1 -- 172.21.15.96:0/1593989841 shutdown_connections 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.270+0000 7f473904a640 1 -- 172.21.15.96:0/1593989841 wait complete. 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.271+0000 7f473904a640 10 monclient: build_initial_monmap 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.271+0000 7f473904a640 10 monclient: monmap: 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:59.275 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.271+0000 7f473904a640 1 librados: starting msgr at 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.271+0000 7f473904a640 1 librados: starting objecter 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 Processor -- start 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 -- start start 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 librados: setting wanted keys 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 librados: calling monclient init 2024-03-25T17:38:59.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient: init 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient(hunting): picked mon.noname-b con 0x7f47340a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient(hunting): picked mon.noname-c con 0x7f473413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient(hunting): picked mon.noname-a con 0x7f47341430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340abdc0 con 0x7f47341430c0 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47341d7730 con 0x7f47340a98f0 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4734138f10 con 0x7f473413f710 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:59.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.272+0000 7f473904a640 10 monclient(hunting): authenticate will time out at 5897.554737s 2024-03-25T17:38:59.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.284+0000 7f4732d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f47340a98f0 0x7f4734138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:52140/0 (socket says 172.21.15.96:52140) 2024-03-25T17:38:59.286 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.284+0000 7f4732d9d640 1 -- 172.21.15.96:0/4022675266 learned_addr learned my addr 172.21.15.96:0/4022675266 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:38:59.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.287+0000 7f471b7fe640 10 client.?.objecter ms_handle_connect 0x7f47340a98f0 2024-03-25T17:38:59.288 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.287+0000 7f471b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:59.289 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.289+0000 7f471b7fe640 10 client.?.objecter ms_handle_connect 0x7f473413f710 2024-03-25T17:38:59.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.289+0000 7f471b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.292+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488090755 0 0) 0x7f47341d7730 con 0x7f47340a98f0 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720004eb0 con 0x7f47340a98f0 2024-03-25T17:38:59.294 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 client.?.objecter ms_handle_connect 0x7f47341430c0 2024-03-25T17:38:59.295 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.293+0000 7f471b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:59.297 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757284133 0 0) 0x7f4734138f10 con 0x7f473413f710 2024-03-25T17:38:59.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 10 monclient(hunting): my global_id is 4661 2024-03-25T17:38:59.298 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.297+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47341d7730 con 0x7f473413f710 2024-03-25T17:38:59.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529483182 0 0) 0x7f4720004eb0 con 0x7f47340a98f0 2024-03-25T17:38:59.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4734138f10 con 0x7f47340a98f0 2024-03-25T17:38:59.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f47240019f0 con 0x7f47340a98f0 2024-03-25T17:38:59.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:59.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 10 monclient(hunting): dump: 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:59.307 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.305+0000 7f471b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:59.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349609817 0 0) 0x7f47340abdc0 con 0x7f47341430c0 2024-03-25T17:38:59.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:38:59.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:59.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:59.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 10 monclient(hunting): my global_id is 4677 2024-03-25T17:38:59.309 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.307+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720004eb0 con 0x7f47341430c0 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1582859590 0 0) 0x7f4734138f10 con 0x7f47340a98f0 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 >> v1:172.21.15.96:6790/0 conn(0x7f473413f710 legacy=0x7f47341d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 >> v1:172.21.15.96:6789/0 conn(0x7f47341430c0 legacy=0x7f47341d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 1 monclient: found mon.b 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:59.323 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47341390a0 con 0x7f47340a98f0 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f471b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 5 monclient: authenticate success, global_id 4657 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 10 monclient: _renew_subs 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47341d8320 con 0x7f47340a98f0 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 10 monclient: _renew_subs 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:59.324 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47341d43c0 con 0x7f47340a98f0 2024-03-25T17:38:59.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.321+0000 7f473904a640 1 librados: init done 2024-03-25T17:38:59.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.322+0000 7f473904a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:59.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.322+0000 7f473904a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:59.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.322+0000 7f473904a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:59.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.322+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340627b0 con 0x7f47340a98f0 2024-03-25T17:38:59.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4724001c50 con 0x7f47340a98f0 2024-03-25T17:38:59.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:59.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f4724004c30 con 0x7f47340a98f0 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f471b7fe640 10 monclient: dump: 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:38:59.615 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:38:59.616 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:38:59.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.613+0000 7f473159a640 4 set_mon_vals no callback set 2024-03-25T17:38:59.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.614+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f472401ac00 con 0x7f47340a98f0 2024-03-25T17:38:59.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.615+0000 7f473259c640 10 monclient: get_auth_request con 0x7f472006de00 auth_method 0 2024-03-25T17:38:59.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.615+0000 7f471b7fe640 10 client.4657.objecter ms_handle_connect 0x7f472006de00 2024-03-25T17:38:59.620 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.620+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3919243714 0 0) 0x7f4724054850 con 0x7f47340a98f0 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.620+0000 7f471b7fe640 10 client.4657.objecter ms_dispatch 0x7f4734000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.620+0000 7f471b7fe640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.620+0000 7f471b7fe640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.620+0000 7f471b7fe640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.621+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4724050360 con 0x7f47340a98f0 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.621+0000 7f471b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:59.621 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.621+0000 7f471b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:59.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.886+0000 7f473904a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:59.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.886+0000 7f473904a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:59.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.886+0000 7f473904a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:38:59.888 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.886+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4734069f40 con 0x7f47340a98f0 2024-03-25T17:38:59.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.976+0000 7f471b7fe640 1 -- 172.21.15.96:0/4022675266 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2732654504) 0x7f4724052290 con 0x7f47340a98f0 2024-03-25T17:38:59.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.976+0000 7f471b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:59.977 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.976+0000 7f471b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:59.977 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:38:59.978 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:29:08.116227+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:08.116227+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:59.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 10 librados: watch_flush enter 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 10 librados: watch_flush exit 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 10 client.4657.objecter successfully canceled tick 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 >> v1:172.21.15.203:6800/3611964061 conn(0x7f472006de00 legacy=0x7f47200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 10 monclient: shutdown 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.978+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 >> v1:172.21.15.203:6789/0 conn(0x7f47340a98f0 legacy=0x7f4734138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.979+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 shutdown_connections 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.979+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 >> 172.21.15.96:0/4022675266 conn(0x7f47340b7ea0 msgr2=0x7f4734074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.979+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 shutdown_connections 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.979+0000 7f473904a640 1 -- 172.21.15.96:0/4022675266 wait complete. 2024-03-25T17:38:59.980 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:38:59.979+0000 7f473904a640 1 librados: shutdown 2024-03-25T17:38:59.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:06.798 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:06.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.896+0000 7fd5b9225640 10 monclient: get_monmap_and_config 2024-03-25T17:39:06.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.896+0000 7fd5b9225640 10 monclient: build_initial_monmap 2024-03-25T17:39:06.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.896+0000 7fd5b9225640 10 monclient: monmap: 2024-03-25T17:39:06.897 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:06.897996+0000 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:06.897996+0000 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:06.898 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.897+0000 7fd5b9225640 1 Processor -- start 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.897+0000 7fd5b9225640 1 -- start start 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.897+0000 7fd5b9225640 10 monclient: init 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:06.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b41430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:06.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b4142520 con 0x7fd5b41430c0 2024-03-25T17:39:06.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40b28a0 con 0x7fd5b40a98f0 2024-03-25T17:39:06.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40b2a30 con 0x7fd5b413f710 2024-03-25T17:39:06.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:06.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.898+0000 7fd5b9225640 10 monclient(hunting): authenticate will time out at 5905.180180s 2024-03-25T17:39:06.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.947+0000 7fd5b2d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd5b40a98f0 0x7fd5b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:37290/0 (socket says 172.21.15.96:37290) 2024-03-25T17:39:06.949 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:06.948+0000 7fd5b2d9d640 1 -- 172.21.15.96:0/3735128689 learned_addr learned my addr 172.21.15.96:0/3735128689 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485660905 0 0) 0x7fd5b4142520 con 0x7fd5b41430c0 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 10 monclient(hunting): my global_id is 4680 2024-03-25T17:39:07.032 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.031+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd59c003520 con 0x7fd5b41430c0 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259797748 0 0) 0x7fd5b40b2a30 con 0x7fd5b413f710 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 10 monclient(hunting): my global_id is 4664 2024-03-25T17:39:07.076 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.075+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b4142520 con 0x7fd5b413f710 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644955837 0 0) 0x7fd5b40b28a0 con 0x7fd5b40a98f0 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 10 monclient(hunting): my global_id is 4660 2024-03-25T17:39:07.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.076+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b40b2a30 con 0x7fd5b40a98f0 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660500518 0 0) 0x7fd59c003520 con 0x7fd5b41430c0 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 >> v1:172.21.15.96:6790/0 conn(0x7fd5b413f710 legacy=0x7fd5b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 >> v1:172.21.15.203:6789/0 conn(0x7fd5b40a98f0 legacy=0x7fd5b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 monclient: found mon.noname-a 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:07.144 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40b44c0 con 0x7fd5b41430c0 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: _finish_auth 0 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fd5ac001a10 con 0x7fd5b41430c0 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b9225640 5 monclient: authenticate success, global_id 4680 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b9225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b1d9b640 10 monclient: dump: 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:07.145 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:07.146 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.143+0000 7fd5b9225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:07.147 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:31.370 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.147 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:31.374 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.174 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:31.400 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.205 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:31.431 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.206 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:31.421 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.208 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:31.435 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.218 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:31.434 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.239 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:31.455 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.241 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:31.468 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5ac001b70 con 0x7fd5b41430c0 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 1 -- 172.21.15.96:0/3735128689 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fd5ac001f80 con 0x7fd5b41430c0 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b9225640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b9225640 4 set_mon_vals no callback set 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.300+0000 7fd5b1d9b640 10 monclient: dump: 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.301 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:07.302 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:07.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 10 monclient: shutdown 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 1 -- 172.21.15.96:0/3735128689 >> v1:172.21.15.96:6789/0 conn(0x7fd5b41430c0 legacy=0x7fd5b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 1 -- 172.21.15.96:0/3735128689 shutdown_connections 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 1 -- 172.21.15.96:0/3735128689 >> 172.21.15.96:0/3735128689 conn(0x7fd5b40b7ea0 msgr2=0x7fd5b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 1 -- 172.21.15.96:0/3735128689 shutdown_connections 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 1 -- 172.21.15.96:0/3735128689 wait complete. 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 10 monclient: build_initial_monmap 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.301+0000 7fd5b9225640 10 monclient: monmap: 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:07.305 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 librados: starting msgr at 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 librados: starting objecter 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 Processor -- start 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 -- start start 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 librados: setting wanted keys 2024-03-25T17:39:07.306 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 1 librados: calling monclient init 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.302+0000 7fd5b9225640 10 monclient: init 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b413f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b4136c50 con 0x7fd5b40a98f0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b4136ec0 con 0x7fd5b41430c0 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b41317f0 con 0x7fd5b413f710 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:07.307 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.303+0000 7fd5b9225640 10 monclient(hunting): authenticate will time out at 5905.585704s 2024-03-25T17:39:07.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.324+0000 7fd5b2d9d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd5b40a98f0 0x7fd5b41310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55104/0 (socket says 172.21.15.96:55104) 2024-03-25T17:39:07.325 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.324+0000 7fd5b2d9d640 1 -- 172.21.15.96:0/4205041696 learned_addr learned my addr 172.21.15.96:0/4205041696 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.325+0000 7fd59b7fe640 10 client.?.objecter ms_handle_connect 0x7fd5b40a98f0 2024-03-25T17:39:07.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.325+0000 7fd59b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:07.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214725805 0 0) 0x7fd5b4136c50 con 0x7fd5b40a98f0 2024-03-25T17:39:07.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:07.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:07.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:07.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 10 monclient(hunting): my global_id is 4683 2024-03-25T17:39:07.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.330+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a0004eb0 con 0x7fd5b40a98f0 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303815343 0 0) 0x7fd5a0004eb0 con 0x7fd5b40a98f0 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b4136c50 con 0x7fd5b40a98f0 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fd5a40019b0 con 0x7fd5b40a98f0 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:07.340 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:07.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.339+0000 7fd59b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:07.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3584726867 0 0) 0x7fd5b4136c50 con 0x7fd5b40a98f0 2024-03-25T17:39:07.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 >> v1:172.21.15.96:6790/0 conn(0x7fd5b413f710 legacy=0x7fd5b41dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 >> v1:172.21.15.203:6789/0 conn(0x7fd5b41430c0 legacy=0x7fd5b41db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:07.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:07.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 1 monclient: found mon.a 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4131980 con 0x7fd5b40a98f0 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd59b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 5 monclient: authenticate success, global_id 4683 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 10 monclient: _renew_subs 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b4135070 con 0x7fd5b40a98f0 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 10 monclient: _renew_subs 2024-03-25T17:39:07.351 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b40b7870 con 0x7fd5b40a98f0 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.348+0000 7fd5b9225640 1 librados: init done 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.349+0000 7fd5b9225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.349+0000 7fd5b9225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.349+0000 7fd5b9225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.352 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.349+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4135c70 con 0x7fd5b40a98f0 2024-03-25T17:39:07.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a4001b10 con 0x7fd5b40a98f0 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fd5a4004c30 con 0x7fd5b40a98f0 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd59b7fe640 10 monclient: dump: 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:07.650 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.648+0000 7fd5b159a640 4 set_mon_vals no callback set 2024-03-25T17:39:07.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.649+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fd5a401ac00 con 0x7fd5b40a98f0 2024-03-25T17:39:07.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.650+0000 7fd5b259c640 10 monclient: get_auth_request con 0x7fd5a006db30 auth_method 0 2024-03-25T17:39:07.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.650+0000 7fd59b7fe640 10 client.4683.objecter ms_handle_connect 0x7fd5a006db30 2024-03-25T17:39:07.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7fd5a40548f0 con 0x7fd5b40a98f0 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 10 client.4683.objecter ms_dispatch 0x7fd5b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.655+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5a4050400 con 0x7fd5b40a98f0 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.656+0000 7fd59b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:07.657 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.656+0000 7fd59b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:07.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.900+0000 7fd5b9225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:07.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.900+0000 7fd5b9225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:07.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.900+0000 7fd5b9225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:07.901 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:07.900+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b40627b0 con 0x7fd5b40a98f0 2024-03-25T17:39:08.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.009+0000 7fd59b7fe640 1 -- 172.21.15.96:0/4205041696 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7fd5a4052330 con 0x7fd5b40a98f0 2024-03-25T17:39:08.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.009+0000 7fd59b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:08.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.009+0000 7fd59b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:08.011 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:08.011 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:08.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 10 librados: watch_flush enter 2024-03-25T17:39:08.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 10 librados: watch_flush exit 2024-03-25T17:39:08.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T17:39:08.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 10 client.4683.objecter successfully canceled tick 2024-03-25T17:39:08.014 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 >> v1:172.21.15.203:6800/3611964061 conn(0x7fd5a006db30 legacy=0x7fd5a006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 10 monclient: shutdown 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 >> v1:172.21.15.96:6789/0 conn(0x7fd5b40a98f0 legacy=0x7fd5b41310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 shutdown_connections 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 >> 172.21.15.96:0/4205041696 conn(0x7fd5b40b7ea0 msgr2=0x7fd5b413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 shutdown_connections 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 -- 172.21.15.96:0/4205041696 wait complete. 2024-03-25T17:39:08.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:08.012+0000 7fd5b9225640 1 librados: shutdown 2024-03-25T17:39:08.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:14.831 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 10 monclient: get_monmap_and_config 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 10 monclient: build_initial_monmap 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 10 monclient: monmap: 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:14.934406+0000 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:14.934406+0000 2024-03-25T17:39:14.934 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:14.935 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:14.935 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:14.935 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:14.935 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:14.935 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 1 Processor -- start 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 1 -- start start 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.934+0000 7fee2ea5c640 10 monclient: init 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-a con 0x7fee280b10d0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-c con 0x7fee280b4360 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-b con 0x7fee280a9470 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280a9ab0 con 0x7fee280b10d0 2024-03-25T17:39:14.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280b2400 con 0x7fee280b4360 2024-03-25T17:39:14.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280aa2a0 con 0x7fee280a9470 2024-03-25T17:39:14.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:14.938 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.935+0000 7fee2ea5c640 10 monclient(hunting): authenticate will time out at 5913.216594s 2024-03-25T17:39:14.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.949+0000 7fee27fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fee280b10d0 0x7fee280b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57022/0 (socket says 172.21.15.96:57022) 2024-03-25T17:39:14.948 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.949+0000 7fee27fff640 1 -- 172.21.15.96:0/2178429391 learned_addr learned my addr 172.21.15.96:0/2178429391 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863682288 0 0) 0x7fee280a9ab0 con 0x7fee280b10d0 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 10 monclient(hunting): my global_id is 4686 2024-03-25T17:39:14.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.956+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee10003520 con 0x7fee280b10d0 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270684369 0 0) 0x7fee10003520 con 0x7fee280b10d0 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 >> v1:172.21.15.96:6790/0 conn(0x7fee280a9470 legacy=0x7fee2814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 >> v1:172.21.15.203:6789/0 conn(0x7fee280b4360 legacy=0x7fee280b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 monclient: found mon.noname-a 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:14.966 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee280b14b0 con 0x7fee280b10d0 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee18001690 con 0x7fee280b10d0 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee26ffd640 10 monclient: dump: 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:14.967 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee2ea5c640 5 monclient: authenticate success, global_id 4686 2024-03-25T17:39:14.968 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.966+0000 7fee2ea5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:14.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee180017f0 con 0x7fee280b10d0 2024-03-25T17:39:14.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:14.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 1 -- 172.21.15.96:0/2178429391 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee18001c40 con 0x7fee280b10d0 2024-03-25T17:39:14.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:14.969 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee26ffd640 10 monclient: dump: 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:14.970 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:14.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee2ea5c640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:14.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.969+0000 7fee2ea5c640 4 set_mon_vals no callback set 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 10 monclient: shutdown 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 1 -- 172.21.15.96:0/2178429391 >> v1:172.21.15.96:6789/0 conn(0x7fee280b10d0 legacy=0x7fee280b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 1 -- 172.21.15.96:0/2178429391 shutdown_connections 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 1 -- 172.21.15.96:0/2178429391 >> 172.21.15.96:0/2178429391 conn(0x7fee280617a0 msgr2=0x7fee28061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:14.973 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 1 -- 172.21.15.96:0/2178429391 shutdown_connections 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.970+0000 7fee2ea5c640 1 -- 172.21.15.96:0/2178429391 wait complete. 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.971+0000 7fee2ea5c640 10 monclient: build_initial_monmap 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.971+0000 7fee2ea5c640 10 monclient: monmap: 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:14.974 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.971+0000 7fee2ea5c640 1 librados: starting msgr at 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.971+0000 7fee2ea5c640 1 librados: starting objecter 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.971+0000 7fee2ea5c640 1 Processor -- start 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 -- start start 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 librados: setting wanted keys 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 librados: calling monclient init 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient: init 2024-03-25T17:39:14.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-c con 0x7fee280a9470 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-a con 0x7fee280b10d0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient(hunting): picked mon.noname-b con 0x7fee280b4360 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2813d520 con 0x7fee280b10d0 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee28136cc0 con 0x7fee280b4360 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2813ece0 con 0x7fee280a9470 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:14.976 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.972+0000 7fee2ea5c640 10 monclient(hunting): authenticate will time out at 5913.253838s 2024-03-25T17:39:14.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.979+0000 7fee27fff640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fee280a9470 0x7fee280ab8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:54266/0 (socket says 172.21.15.96:54266) 2024-03-25T17:39:14.979 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.979+0000 7fee27fff640 1 -- 172.21.15.96:0/3289097100 learned_addr learned my addr 172.21.15.96:0/3289097100 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.981+0000 7fee24ff9640 10 client.?.objecter ms_handle_connect 0x7fee280a9470 2024-03-25T17:39:14.981 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.981+0000 7fee24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.992+0000 7fee24ff9640 10 client.?.objecter ms_handle_connect 0x7fee280b4360 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.992+0000 7fee24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.992+0000 7fee24ff9640 10 client.?.objecter ms_handle_connect 0x7fee280b10d0 2024-03-25T17:39:14.992 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.992+0000 7fee24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:14.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762736775 0 0) 0x7fee2813ece0 con 0x7fee280a9470 2024-03-25T17:39:14.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 10 monclient(hunting): my global_id is 4670 2024-03-25T17:39:14.994 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:14.994+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee14004eb0 con 0x7fee280a9470 2024-03-25T17:39:15.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235493246 0 0) 0x7fee28136cc0 con 0x7fee280b4360 2024-03-25T17:39:15.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:15.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:15.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:15.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 10 monclient(hunting): my global_id is 4666 2024-03-25T17:39:15.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.010+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee2813ece0 con 0x7fee280b4360 2024-03-25T17:39:15.011 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298390533 0 0) 0x7fee2813d520 con 0x7fee280b10d0 2024-03-25T17:39:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 10 monclient(hunting): my global_id is 4689 2024-03-25T17:39:15.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.012+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee28136cc0 con 0x7fee280b10d0 2024-03-25T17:39:15.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804924909 0 0) 0x7fee14004eb0 con 0x7fee280a9470 2024-03-25T17:39:15.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee2813d520 con 0x7fee280a9470 2024-03-25T17:39:15.015 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee18001bc0 con 0x7fee280a9470 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:15.016 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:15.017 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:15.017 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.015+0000 7fee24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1561089640 0 0) 0x7fee2813d520 con 0x7fee280a9470 2024-03-25T17:39:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 >> v1:172.21.15.203:6789/0 conn(0x7fee280b4360 legacy=0x7fee281365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 >> v1:172.21.15.96:6789/0 conn(0x7fee280b10d0 legacy=0x7fee28135ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:39:15.033 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 1 monclient: found mon.c 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2813efc0 con 0x7fee280a9470 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 10 monclient: _finish_auth 0 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 5 monclient: authenticate success, global_id 4670 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 10 monclient: _renew_subs 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2813f2a0 con 0x7fee280a9470 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 10 monclient: _renew_subs 2024-03-25T17:39:15.034 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2813d100 con 0x7fee280a9470 2024-03-25T17:39:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.033+0000 7fee2ea5c640 1 librados: init done 2024-03-25T17:39:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.034+0000 7fee2ea5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.035 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.034+0000 7fee2ea5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.034+0000 7fee2ea5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:15.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.034+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee280627b0 con 0x7fee280a9470 2024-03-25T17:39:15.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee18004bf0 con 0x7fee280a9470 2024-03-25T17:39:15.379 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fee18004f00 con 0x7fee280a9470 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee24ff9640 10 monclient: dump: 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:15.380 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:15.381 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:15.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.379+0000 7fee267fc640 4 set_mon_vals no callback set 2024-03-25T17:39:15.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.380+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fee1801aed0 con 0x7fee280a9470 2024-03-25T17:39:15.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.381+0000 7fee277fe640 10 monclient: get_auth_request con 0x7fee1406de20 auth_method 0 2024-03-25T17:39:15.381 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.381+0000 7fee24ff9640 10 client.4670.objecter ms_handle_connect 0x7fee1406de20 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7fee180549c0 con 0x7fee280a9470 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 10 client.4670.objecter ms_dispatch 0x7fee28000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee180504d0 con 0x7fee280a9470 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:15.386 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.386+0000 7fee24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:15.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.646+0000 7fee2ea5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.645 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.646+0000 7fee2ea5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.646+0000 7fee2ea5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:15.646 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.646+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee280b6d90 con 0x7fee280a9470 2024-03-25T17:39:15.679 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.679+0000 7fee24ff9640 1 -- 172.21.15.96:0/3289097100 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7fee18052400 con 0x7fee280a9470 2024-03-25T17:39:15.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.679+0000 7fee24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:15.680 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.679+0000 7fee24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:15.680 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:15.680 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:15.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 10 librados: watch_flush enter 2024-03-25T17:39:15.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 10 librados: watch_flush exit 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 10 client.4670.objecter successfully canceled tick 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 >> v1:172.21.15.203:6800/3611964061 conn(0x7fee1406de20 legacy=0x7fee140702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 10 monclient: shutdown 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 >> v1:172.21.15.96:6790/0 conn(0x7fee280a9470 legacy=0x7fee280ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 shutdown_connections 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 >> 172.21.15.96:0/3289097100 conn(0x7fee280617a0 msgr2=0x7fee280b6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 shutdown_connections 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 -- 172.21.15.96:0/3289097100 wait complete. 2024-03-25T17:39:15.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:15.682+0000 7fee2ea5c640 1 librados: shutdown 2024-03-25T17:39:15.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:22.499 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.614+0000 7f43ed001640 10 monclient: get_monmap_and_config 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.614+0000 7f43ed001640 10 monclient: build_initial_monmap 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.614+0000 7f43ed001640 10 monclient: monmap: 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:22.615029+0000 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:22.615029+0000 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.614 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.615+0000 7f43ed001640 1 Processor -- start 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.615+0000 7f43ed001640 1 -- start start 2024-03-25T17:39:22.615 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.615+0000 7f43ed001640 10 monclient: init 2024-03-25T17:39:22.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:22.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:22.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-a con 0x7f43e80b4a80 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:22.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-c con 0x7f43e813c270 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:22.616 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-b con 0x7f43e80a9930 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:22.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80aaef0 con 0x7f43e80b4a80 2024-03-25T17:39:22.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80ab1d0 con 0x7f43e813c270 2024-03-25T17:39:22.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e813fc80 con 0x7f43e80a9930 2024-03-25T17:39:22.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:22.617 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.616+0000 7f43ed001640 10 monclient(hunting): authenticate will time out at 5920.897510s 2024-03-25T17:39:22.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.625+0000 7f43e6d9d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f43e80b4a80 0x7f43e813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34046/0 (socket says 172.21.15.96:34046) 2024-03-25T17:39:22.625 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.625+0000 7f43e6d9d640 1 -- 172.21.15.96:0/2912385817 learned_addr learned my addr 172.21.15.96:0/2912385817 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897081974 0 0) 0x7f43e80aaef0 con 0x7f43e80b4a80 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): my global_id is 4692 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4003520 con 0x7f43e80b4a80 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255855610 0 0) 0x7f43e813fc80 con 0x7f43e80a9930 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 10 monclient(hunting): my global_id is 4673 2024-03-25T17:39:22.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.636+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e80aaef0 con 0x7f43e80a9930 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523048011 0 0) 0x7f43e80ab1d0 con 0x7f43e813c270 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 10 monclient(hunting): my global_id is 4669 2024-03-25T17:39:22.644 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.644+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e813fc80 con 0x7f43e813c270 2024-03-25T17:39:22.647 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010385537 0 0) 0x7f43d4003520 con 0x7f43e80b4a80 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 >> v1:172.21.15.96:6790/0 conn(0x7f43e80a9930 legacy=0x7f43e8142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 >> v1:172.21.15.203:6789/0 conn(0x7f43e813c270 legacy=0x7f43e8142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: _finish_hunting 0 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 monclient: found mon.noname-a 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80a9ec0 con 0x7f43e80b4a80 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: _finish_auth 0 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: _check_auth_tickets 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43ed001640 5 monclient: authenticate success, global_id 4692 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43ed001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43d8001690 con 0x7f43e80b4a80 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.648 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43e659c640 10 monclient: dump: 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.647+0000 7f43ed001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d80017f0 con 0x7f43e80b4a80 2024-03-25T17:39:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 1 -- 172.21.15.96:0/2912385817 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43d8001c00 con 0x7f43e80b4a80 2024-03-25T17:39:22.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43e659c640 10 monclient: dump: 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43ed001640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:22.651 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43ed001640 4 set_mon_vals no callback set 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.651+0000 7f43ed001640 10 monclient: shutdown 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 1 -- 172.21.15.96:0/2912385817 >> v1:172.21.15.96:6789/0 conn(0x7f43e80b4a80 legacy=0x7f43e813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 1 -- 172.21.15.96:0/2912385817 shutdown_connections 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 1 -- 172.21.15.96:0/2912385817 >> 172.21.15.96:0/2912385817 conn(0x7f43e80615c0 msgr2=0x7f43e80619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 1 -- 172.21.15.96:0/2912385817 shutdown_connections 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.652+0000 7f43ed001640 1 -- 172.21.15.96:0/2912385817 wait complete. 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 10 monclient: build_initial_monmap 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 10 monclient: monmap: 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:22.653 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 librados: starting msgr at 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 librados: starting objecter 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 Processor -- start 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 -- start start 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 librados: setting wanted keys 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 1 librados: calling monclient init 2024-03-25T17:39:22.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.653+0000 7f43ed001640 10 monclient: init 2024-03-25T17:39:22.655 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:22.655 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:22.655 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-b con 0x7f43e80a9930 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-a con 0x7f43e80b4a80 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient(hunting): picked mon.noname-c con 0x7f43e813c270 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e81366b0 con 0x7f43e80b4a80 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e81457b0 con 0x7f43e80a9930 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e81453b0 con 0x7f43e813c270 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:22.656 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.654+0000 7f43ed001640 10 monclient(hunting): authenticate will time out at 5920.935276s 2024-03-25T17:39:22.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.669+0000 7f43dffff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f43e80b4a80 0x7f43e8131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34070/0 (socket says 172.21.15.96:34070) 2024-03-25T17:39:22.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.669+0000 7f43dffff640 1 -- 172.21.15.96:0/2741156036 learned_addr learned my addr 172.21.15.96:0/2741156036 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:22.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.669+0000 7f43df7fe640 10 client.?.objecter ms_handle_connect 0x7f43e80b4a80 2024-03-25T17:39:22.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.669+0000 7f43df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.672+0000 7f43df7fe640 10 client.?.objecter ms_handle_connect 0x7f43e813c270 2024-03-25T17:39:22.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.672+0000 7f43df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.682 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.683+0000 7f43df7fe640 10 client.?.objecter ms_handle_connect 0x7f43e80a9930 2024-03-25T17:39:22.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.683+0000 7f43df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:22.689 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311760302 0 0) 0x7f43e81366b0 con 0x7f43e80b4a80 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.689+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c8004eb0 con 0x7f43e80b4a80 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094219984 0 0) 0x7f43e81453b0 con 0x7f43e813c270 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T17:39:22.690 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.690+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e81366b0 con 0x7f43e813c270 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904763761 0 0) 0x7f43e81457b0 con 0x7f43e80a9930 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T17:39:22.695 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.695+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e81453b0 con 0x7f43e80a9930 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433848238 0 0) 0x7f43e81366b0 con 0x7f43e813c270 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e81457b0 con 0x7f43e813c270 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43e0001cb0 con 0x7f43e813c270 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:22.709 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.710 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.709+0000 7f43df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460744196 0 0) 0x7f43c8004eb0 con 0x7f43e80b4a80 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e81366b0 con 0x7f43e80b4a80 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43d0001930 con 0x7f43e80b4a80 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.710+0000 7f43df7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:22.711 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:22.712 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869780356 0 0) 0x7f43e81453b0 con 0x7f43e80a9930 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43c8004eb0 con 0x7f43e80a9930 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43d8001b00 con 0x7f43e80a9930 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.720+0000 7f43df7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:22.720 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:22.721 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:22.721 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:22.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1410580941 0 0) 0x7f43e81366b0 con 0x7f43e80b4a80 2024-03-25T17:39:22.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 >> v1:172.21.15.96:6790/0 conn(0x7f43e813c270 legacy=0x7f43e81da860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 >> v1:172.21.15.203:6789/0 conn(0x7f43e80a9930 legacy=0x7f43e8144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:22.724 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 1 monclient: found mon.a 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8145540 con 0x7f43e80b4a80 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43ed001640 5 monclient: authenticate success, global_id 4695 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43ed001640 10 monclient: _renew_subs 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43ed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.724+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e8132620 con 0x7f43e80b4a80 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.725+0000 7f43ed001640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.725+0000 7f43ed001640 10 monclient: _renew_subs 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.725+0000 7f43ed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:22.725 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.725+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e813e660 con 0x7f43e80b4a80 2024-03-25T17:39:22.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.725+0000 7f43ed001640 1 librados: init done 2024-03-25T17:39:22.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.726+0000 7f43ed001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:22.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.726+0000 7f43ed001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:22.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.726+0000 7f43ed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:22.726 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:22.726+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e80aa420 con 0x7f43e80b4a80 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d0003610 con 0x7f43e80b4a80 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f43d00039a0 con 0x7f43e80b4a80 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43df7fe640 10 monclient: dump: 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:23.019 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:23.020 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.018+0000 7f43e5d9b640 4 set_mon_vals no callback set 2024-03-25T17:39:23.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.019+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f43d0003bc0 con 0x7f43e80b4a80 2024-03-25T17:39:23.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.019+0000 7f43e6d9d640 10 monclient: get_auth_request con 0x7f43c806e210 auth_method 0 2024-03-25T17:39:23.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.019+0000 7f43df7fe640 10 client.4695.objecter ms_handle_connect 0x7f43c806e210 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f43d0054730 con 0x7f43e80b4a80 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 10 client.4695.objecter ms_dispatch 0x7f43e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43d0050240 con 0x7f43e80b4a80 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:23.025 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.024+0000 7f43df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:23.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.281+0000 7f43ed001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.281+0000 7f43ed001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.281+0000 7f43ed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:23.282 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.281+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43e813e7f0 con 0x7f43e80b4a80 2024-03-25T17:39:23.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.354+0000 7f43df7fe640 1 -- 172.21.15.96:0/2741156036 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f43d0052170 con 0x7f43e80b4a80 2024-03-25T17:39:23.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.354+0000 7f43df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:23.355 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.354+0000 7f43df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:23.355 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:23.355 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 10 librados: watch_flush enter 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 10 librados: watch_flush exit 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 10 client.4695.objecter successfully canceled tick 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 >> v1:172.21.15.203:6800/3611964061 conn(0x7f43c806e210 legacy=0x7f43c80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 10 monclient: shutdown 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:23.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 >> v1:172.21.15.96:6789/0 conn(0x7f43e80b4a80 legacy=0x7f43e8131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:23.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 shutdown_connections 2024-03-25T17:39:23.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 >> 172.21.15.96:0/2741156036 conn(0x7f43e80615c0 msgr2=0x7f43e8142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:23.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 shutdown_connections 2024-03-25T17:39:23.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 -- 172.21.15.96:0/2741156036 wait complete. 2024-03-25T17:39:23.359 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:23.357+0000 7f43ed001640 1 librados: shutdown 2024-03-25T17:39:23.370 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:27.077 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:51.304 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.084 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:51.311 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.126 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:10:51.353 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.136 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:51.351 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.144 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:51.359 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.155 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:51.382 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.165 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:51.392 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.167 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:51.382 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:27.192 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:10:51.419 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:30.175 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.289+0000 7fe4e967b640 10 monclient: get_monmap_and_config 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.289+0000 7fe4e967b640 10 monclient: build_initial_monmap 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.289+0000 7fe4e967b640 10 monclient: monmap: 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:30.290659+0000 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:30.290659+0000 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:30.290 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.291 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 1 Processor -- start 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 1 -- start start 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient: init 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e40a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e413f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e41430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4142520 con 0x7fe4e40a98f0 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e40b28a0 con 0x7fe4e413f710 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e40b2a30 con 0x7fe4e41430c0 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:30.292 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.290+0000 7fe4e967b640 10 monclient(hunting): authenticate will time out at 5928.572920s 2024-03-25T17:39:30.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.301+0000 7fe4e2ffd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe4e40a98f0 0x7fe4e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33502/0 (socket says 172.21.15.96:33502) 2024-03-25T17:39:30.302 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.301+0000 7fe4e2ffd640 1 -- 172.21.15.96:0/1966317956 learned_addr learned my addr 172.21.15.96:0/1966317956 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427097884 0 0) 0x7fe4e4142520 con 0x7fe4e40a98f0 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 10 monclient(hunting): my global_id is 4698 2024-03-25T17:39:30.314 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.313+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4cc003520 con 0x7fe4e40a98f0 2024-03-25T17:39:30.315 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445901809 0 0) 0x7fe4e40b2a30 con 0x7fe4e41430c0 2024-03-25T17:39:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 10 monclient(hunting): my global_id is 4679 2024-03-25T17:39:30.316 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.315+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4142520 con 0x7fe4e41430c0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548895319 0 0) 0x7fe4cc003520 con 0x7fe4e40a98f0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 >> v1:172.21.15.96:6790/0 conn(0x7fe4e41430c0 legacy=0x7fe4e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 >> v1:172.21.15.203:6789/0 conn(0x7fe4e413f710 legacy=0x7fe4e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 1 monclient: found mon.noname-a 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e40b44c0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:39:30.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:39:30.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:30.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e967b640 5 monclient: authenticate success, global_id 4698 2024-03-25T17:39:30.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.320+0000 7fe4e967b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe4d4001650 con 0x7fe4e40a98f0 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: dump: 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:30.326 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4d40018a0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 1 -- 172.21.15.96:0/1966317956 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe4d4001c30 con 0x7fe4e40a98f0 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e1ffb640 10 monclient: dump: 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.327 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:30.328 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:30.328 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:30.328 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e967b640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:30.328 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.325+0000 7fe4e967b640 4 set_mon_vals no callback set 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 10 monclient: shutdown 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 1 -- 172.21.15.96:0/1966317956 >> v1:172.21.15.96:6789/0 conn(0x7fe4e40a98f0 legacy=0x7fe4e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 1 -- 172.21.15.96:0/1966317956 shutdown_connections 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 1 -- 172.21.15.96:0/1966317956 >> 172.21.15.96:0/1966317956 conn(0x7fe4e40b7ea0 msgr2=0x7fe4e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 1 -- 172.21.15.96:0/1966317956 shutdown_connections 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.326+0000 7fe4e967b640 1 -- 172.21.15.96:0/1966317956 wait complete. 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.327+0000 7fe4e967b640 10 monclient: build_initial_monmap 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.327+0000 7fe4e967b640 10 monclient: monmap: 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:30.330 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.327+0000 7fe4e967b640 1 librados: starting msgr at 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.327+0000 7fe4e967b640 1 librados: starting objecter 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.328+0000 7fe4e967b640 1 Processor -- start 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.328+0000 7fe4e967b640 1 -- start start 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.328+0000 7fe4e967b640 1 librados: setting wanted keys 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.328+0000 7fe4e967b640 1 librados: calling monclient init 2024-03-25T17:39:30.331 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.328+0000 7fe4e967b640 10 monclient: init 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e40a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e413f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4136d30 con 0x7fe4e413f710 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4137010 con 0x7fe4e41430c0 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e41372f0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:30.332 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.329+0000 7fe4e967b640 10 monclient(hunting): authenticate will time out at 5928.611555s 2024-03-25T17:39:30.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.333+0000 7fe4e27fc640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe4e413f710 0x7fe4e4135370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33538/0 (socket says 172.21.15.96:33538) 2024-03-25T17:39:30.333 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.333+0000 7fe4e27fc640 1 -- 172.21.15.96:0/2989344302 learned_addr learned my addr 172.21.15.96:0/2989344302 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:30.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.333+0000 7fe4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fe4e413f710 2024-03-25T17:39:30.334 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.333+0000 7fe4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.337+0000 7fe4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fe4e40a98f0 2024-03-25T17:39:30.337 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.337+0000 7fe4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164532838 0 0) 0x7fe4e4136d30 con 0x7fe4e413f710 2024-03-25T17:39:30.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.341 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 10 monclient(hunting): my global_id is 4701 2024-03-25T17:39:30.342 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.341+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0004eb0 con 0x7fe4e413f710 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78960592 0 0) 0x7fe4e41372f0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 10 monclient(hunting): my global_id is 4682 2024-03-25T17:39:30.349 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.348+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4136d30 con 0x7fe4e40a98f0 2024-03-25T17:39:30.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.349+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3759409202 0 0) 0x7fe4d0004eb0 con 0x7fe4e413f710 2024-03-25T17:39:30.350 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.349+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4e41372f0 con 0x7fe4e413f710 2024-03-25T17:39:30.353 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.353+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe4d8001a70 con 0x7fe4e413f710 2024-03-25T17:39:30.353 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.353+0000 7fe4cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.353 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.353+0000 7fe4cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.353+0000 7fe4cb7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.354 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.353+0000 7fe4cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.356+0000 7fe4cb7fe640 10 client.?.objecter ms_handle_connect 0x7fe4e41430c0 2024-03-25T17:39:30.357 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.356+0000 7fe4cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604490063 0 0) 0x7fe4e4136d30 con 0x7fe4e40a98f0 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d0004eb0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe4d4001bb0 con 0x7fe4e40a98f0 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.357+0000 7fe4cb7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:30.358 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:30.359 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 98713764 0 0) 0x7fe4e41372f0 con 0x7fe4e413f710 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 >> v1:172.21.15.96:6790/0 conn(0x7fe4e40a98f0 legacy=0x7fe4e41379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 >> v1:172.21.15.203:6789/0 conn(0x7fe4e41430c0 legacy=0x7fe4e40b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 1 monclient: found mon.a 2024-03-25T17:39:30.365 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e41375d0 con 0x7fe4e413f710 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 5 monclient: authenticate success, global_id 4701 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 10 monclient: _renew_subs 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e4135c00 con 0x7fe4e413f710 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 10 monclient: _renew_subs 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e41e3690 con 0x7fe4e413f710 2024-03-25T17:39:30.366 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.364+0000 7fe4e967b640 1 librados: init done 2024-03-25T17:39:30.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.365+0000 7fe4e967b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.365+0000 7fe4e967b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.365+0000 7fe4e967b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.367 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.365+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e40627b0 con 0x7fe4e413f710 2024-03-25T17:39:30.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.585+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4d8001bd0 con 0x7fe4e413f710 2024-03-25T17:39:30.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.585+0000 7fe4cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:30.585 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.585+0000 7fe4e17fa640 4 set_mon_vals no callback set 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.669+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fe4d8003ae0 con 0x7fe4e413f710 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.669+0000 7fe4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.669+0000 7fe4cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.669+0000 7fe4cb7fe640 10 monclient: dump: 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:30.670 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.670+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fe4d8003cc0 con 0x7fe4e413f710 2024-03-25T17:39:30.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.671+0000 7fe4e2ffd640 10 monclient: get_auth_request con 0x7fe4d006de10 auth_method 0 2024-03-25T17:39:30.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.671+0000 7fe4cb7fe640 10 client.4701.objecter ms_handle_connect 0x7fe4d006de10 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7fe4d8054920 con 0x7fe4e413f710 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 10 client.4701.objecter ms_dispatch 0x7fe4e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4d8050260 con 0x7fe4e413f710 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:30.678 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.677+0000 7fe4cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:30.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.910+0000 7fe4e967b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.910+0000 7fe4e967b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.910+0000 7fe4e967b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:30.911 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.910+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4e413df50 con 0x7fe4e413f710 2024-03-25T17:39:30.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.928+0000 7fe4cb7fe640 1 -- 172.21.15.96:0/2989344302 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7fe4d8001d40 con 0x7fe4e413f710 2024-03-25T17:39:30.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.928+0000 7fe4cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:30.928 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.928+0000 7fe4cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:30.929 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:30.929 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:30.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 10 librados: watch_flush enter 2024-03-25T17:39:30.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 10 librados: watch_flush exit 2024-03-25T17:39:30.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T17:39:30.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 10 client.4701.objecter successfully canceled tick 2024-03-25T17:39:30.931 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 >> v1:172.21.15.203:6800/3611964061 conn(0x7fe4d006de10 legacy=0x7fe4d00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 10 monclient: shutdown 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 >> v1:172.21.15.96:6789/0 conn(0x7fe4e413f710 legacy=0x7fe4e4135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 shutdown_connections 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 >> 172.21.15.96:0/2989344302 conn(0x7fe4e40b7ea0 msgr2=0x7fe4e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 shutdown_connections 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 -- 172.21.15.96:0/2989344302 wait complete. 2024-03-25T17:39:30.932 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:30.930+0000 7fe4e967b640 1 librados: shutdown 2024-03-25T17:39:30.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:37.748 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.863+0000 7f181fa2a640 10 monclient: get_monmap_and_config 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.863+0000 7f181fa2a640 10 monclient: build_initial_monmap 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.863+0000 7f181fa2a640 10 monclient: monmap: 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:37.864664+0000 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:37.864664+0000 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:37.864 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.865 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.864+0000 7f181fa2a640 1 Processor -- start 2024-03-25T17:39:37.865 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.864+0000 7f181fa2a640 1 -- start start 2024-03-25T17:39:37.865 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.864+0000 7f181fa2a640 10 monclient: init 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-b con 0x7f18180a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-a con 0x7f181813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-c con 0x7f18181430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1818142520 con 0x7f181813f710 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180b28a0 con 0x7f18181430c0 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180b2a30 con 0x7f18180a98f0 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:37.866 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.865+0000 7f181fa2a640 10 monclient(hunting): authenticate will time out at 5936.147297s 2024-03-25T17:39:37.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.867+0000 7f181d7c6640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f18180a98f0 0x7f18180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:33020/0 (socket says 172.21.15.96:33020) 2024-03-25T17:39:37.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.867+0000 7f181d7c6640 1 -- 172.21.15.96:0/159602103 learned_addr learned my addr 172.21.15.96:0/159602103 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:37.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414758356 0 0) 0x7f18180b2a30 con 0x7f18180a98f0 2024-03-25T17:39:37.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 10 monclient(hunting): my global_id is 4685 2024-03-25T17:39:37.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.872+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1804003520 con 0x7f18180a98f0 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171940890 0 0) 0x7f18180b28a0 con 0x7f18181430c0 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 10 monclient(hunting): my global_id is 4681 2024-03-25T17:39:37.880 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.879+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18180b2a30 con 0x7f18181430c0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161102217 0 0) 0x7f1804003520 con 0x7f18180a98f0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 >> v1:172.21.15.203:6789/0 conn(0x7f18181430c0 legacy=0x7f181814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 >> v1:172.21.15.96:6789/0 conn(0x7f181813f710 legacy=0x7f18181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 monclient: found mon.noname-b 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18180b44c0 con 0x7f18180a98f0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: _finish_auth 0 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:37.885 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1814001690 con 0x7f18180a98f0 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f180ffff640 10 monclient: dump: 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:37.886 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f181fa2a640 5 monclient: authenticate success, global_id 4685 2024-03-25T17:39:37.887 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.884+0000 7f181fa2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18140017f0 con 0x7f18180a98f0 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 1 -- 172.21.15.96:0/159602103 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1814001c00 con 0x7f18180a98f0 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f180ffff640 10 monclient: dump: 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.894 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f181fa2a640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:37.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.893+0000 7f181fa2a640 4 set_mon_vals no callback set 2024-03-25T17:39:37.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.894+0000 7f181fa2a640 10 monclient: shutdown 2024-03-25T17:39:37.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.894+0000 7f181fa2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:37.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.894+0000 7f181fa2a640 1 -- 172.21.15.96:0/159602103 >> v1:172.21.15.96:6790/0 conn(0x7f18180a98f0 legacy=0x7f18180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.894+0000 7f181fa2a640 1 -- 172.21.15.96:0/159602103 shutdown_connections 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.894+0000 7f181fa2a640 1 -- 172.21.15.96:0/159602103 >> 172.21.15.96:0/159602103 conn(0x7f18180b7ea0 msgr2=0x7f18180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.895+0000 7f181fa2a640 1 -- 172.21.15.96:0/159602103 shutdown_connections 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.895+0000 7f181fa2a640 1 -- 172.21.15.96:0/159602103 wait complete. 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.895+0000 7f181fa2a640 10 monclient: build_initial_monmap 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.895+0000 7f181fa2a640 10 monclient: monmap: 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:37.898 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 librados: starting msgr at 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 librados: starting objecter 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 Processor -- start 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 -- start start 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 librados: setting wanted keys 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 librados: calling monclient init 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient: init 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-c con 0x7f18180a98f0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-b con 0x7f181813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient(hunting): picked mon.noname-a con 0x7f18181430c0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18181db7d0 con 0x7f18181430c0 2024-03-25T17:39:37.899 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180b1420 con 0x7f181813f710 2024-03-25T17:39:37.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180b15b0 con 0x7f18180a98f0 2024-03-25T17:39:37.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:37.900 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.896+0000 7f181fa2a640 10 monclient(hunting): authenticate will time out at 5936.178739s 2024-03-25T17:39:37.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.907+0000 7f181dfc7640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f18181430c0 0x7f18181d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58376/0 (socket says 172.21.15.96:58376) 2024-03-25T17:39:37.907 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.907+0000 7f181dfc7640 1 -- 172.21.15.96:0/3660846650 learned_addr learned my addr 172.21.15.96:0/3660846650 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:37.908 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.908+0000 7f180dffb640 10 client.?.objecter ms_handle_connect 0x7f18180a98f0 2024-03-25T17:39:37.908 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.908+0000 7f180dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.909 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.909+0000 7f180dffb640 10 client.?.objecter ms_handle_connect 0x7f18181430c0 2024-03-25T17:39:37.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.909+0000 7f180dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105441682 0 0) 0x7f18180b15b0 con 0x7f18180a98f0 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 10 monclient(hunting): my global_id is 4688 2024-03-25T17:39:37.914 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.913+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17fc004eb0 con 0x7f18180a98f0 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979569544 0 0) 0x7f18181db7d0 con 0x7f18181430c0 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T17:39:37.919 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.918+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18180b15b0 con 0x7f18181430c0 2024-03-25T17:39:37.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.922+0000 7f180dffb640 10 client.?.objecter ms_handle_connect 0x7f181813f710 2024-03-25T17:39:37.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.922+0000 7f180dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942391684 0 0) 0x7f18180b15b0 con 0x7f18181430c0 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18181db7d0 con 0x7f18181430c0 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1810001d20 con 0x7f18181430c0 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 10 monclient(hunting): dump: 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:37.933 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:37.934 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:37.934 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:37.934 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.932+0000 7f180dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:37.934 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3138488529 0 0) 0x7f17fc004eb0 con 0x7f18180a98f0 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18180b15b0 con 0x7f18180a98f0 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1814001b00 con 0x7f18180a98f0 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.933+0000 7f180dffb640 10 monclient(hunting): dump: 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:37.935 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415979598 0 0) 0x7f18180b1420 con 0x7f181813f710 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T17:39:37.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.936+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17fc004eb0 con 0x7f181813f710 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325742268 0 0) 0x7f18181db7d0 con 0x7f18181430c0 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 >> v1:172.21.15.96:6790/0 conn(0x7f18180a98f0 legacy=0x7f18181daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 >> v1:172.21.15.203:6789/0 conn(0x7f181813f710 legacy=0x7f18180b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 1 monclient: found mon.a 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18180b1740 con 0x7f18181430c0 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 10 monclient: _finish_auth 0 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f180dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:37.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 5 monclient: authenticate success, global_id 4707 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 10 monclient: _renew_subs 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18181db1a0 con 0x7f18181430c0 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 10 monclient: _renew_subs 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18180b08f0 con 0x7f18181430c0 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.939+0000 7f181fa2a640 1 librados: init done 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.940+0000 7f181fa2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.940+0000 7f181fa2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:37.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.940+0000 7f181fa2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:37.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:37.940+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18180627b0 con 0x7f18181430c0 2024-03-25T17:39:38.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1810003990 con 0x7f18181430c0 2024-03-25T17:39:38.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:38.238 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1810003ce0 con 0x7f18181430c0 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 10 monclient: dump: 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:38.239 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180f7fe640 4 set_mon_vals no callback set 2024-03-25T17:39:38.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.238+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1810003f00 con 0x7f18181430c0 2024-03-25T17:39:38.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.239+0000 7f181d7c6640 10 monclient: get_auth_request con 0x7f17fc06df00 auth_method 0 2024-03-25T17:39:38.240 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.239+0000 7f180dffb640 10 client.4707.objecter ms_handle_connect 0x7f17fc06df00 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f1810054ac0 con 0x7f18181430c0 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 10 client.4707.objecter ms_dispatch 0x7f1818000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18100505d0 con 0x7f18181430c0 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:38.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.245+0000 7f180dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:38.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.493+0000 7f181fa2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.493+0000 7f181fa2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.493+0000 7f181fa2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:39:38.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.493+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18180abb10 con 0x7f18181430c0 2024-03-25T17:39:38.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.515+0000 7f180dffb640 1 -- 172.21.15.96:0/3660846650 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f1810052500 con 0x7f18181430c0 2024-03-25T17:39:38.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.516+0000 7f180dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:38.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.516+0000 7f180dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:38.517 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:38.517 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.517+0000 7f181fa2a640 10 librados: watch_flush enter 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.517+0000 7f181fa2a640 10 librados: watch_flush exit 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.517+0000 7f181fa2a640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.517+0000 7f181fa2a640 10 client.4707.objecter successfully canceled tick 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 >> v1:172.21.15.203:6800/3611964061 conn(0x7f17fc06df00 legacy=0x7f17fc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 10 monclient: shutdown 2024-03-25T17:39:38.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 >> v1:172.21.15.96:6789/0 conn(0x7f18181430c0 legacy=0x7f18181d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 shutdown_connections 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 >> 172.21.15.96:0/3660846650 conn(0x7f18180b7ea0 msgr2=0x7f181813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 shutdown_connections 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 -- 172.21.15.96:0/3660846650 wait complete. 2024-03-25T17:39:38.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:38.518+0000 7f181fa2a640 1 librados: shutdown 2024-03-25T17:39:38.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:45.337 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:45.464 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.463+0000 7f6e13c7b640 10 monclient: get_monmap_and_config 2024-03-25T17:39:45.464 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.463+0000 7f6e13c7b640 10 monclient: build_initial_monmap 2024-03-25T17:39:45.464 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.463+0000 7f6e13c7b640 10 monclient: monmap: 2024-03-25T17:39:45.464 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:45.464 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:45.464954+0000 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:45.464954+0000 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:45.465 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.464+0000 7f6e13c7b640 1 Processor -- start 2024-03-25T17:39:45.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.464+0000 7f6e13c7b640 1 -- start start 2024-03-25T17:39:45.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.464+0000 7f6e13c7b640 10 monclient: init 2024-03-25T17:39:45.466 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e0c06ab80 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e0c06af60 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e0c0b00b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c0b0500 con 0x7f6e0c0b00b0 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c13ffa0 con 0x7f6e0c06af60 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c0af520 con 0x7f6e0c06ab80 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:45.467 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.465+0000 7f6e13c7b640 10 monclient(hunting): authenticate will time out at 5943.747668s 2024-03-25T17:39:45.472 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.471+0000 7f6e11216640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e0c06af60 0x7f6e0c13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:45790/0 (socket says 172.21.15.96:45790) 2024-03-25T17:39:45.472 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.471+0000 7f6e11216640 1 -- 172.21.15.96:0/3501197135 learned_addr learned my addr 172.21.15.96:0/3501197135 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887539796 0 0) 0x7f6e0c0b0500 con 0x7f6e0c0b00b0 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 10 monclient(hunting): my global_id is 4710 2024-03-25T17:39:45.484 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.483+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df8003520 con 0x7f6e0c0b00b0 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020778327 0 0) 0x7f6e0c13ffa0 con 0x7f6e0c06af60 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 10 monclient(hunting): my global_id is 4687 2024-03-25T17:39:45.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.486+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c0b0500 con 0x7f6e0c06af60 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711496405 0 0) 0x7f6e0c0b0500 con 0x7f6e0c06af60 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 >> v1:172.21.15.96:6790/0 conn(0x7f6e0c06ab80 legacy=0x7f6e0c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 >> v1:172.21.15.96:6789/0 conn(0x7f6e0c0b00b0 legacy=0x7f6e0c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: _finish_hunting 0 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 monclient: found mon.noname-c 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:45.492 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c072890 con 0x7f6e0c06af60 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: _finish_auth 0 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: _check_auth_tickets 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e08001670 con 0x7f6e0c06af60 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e13c7b640 5 monclient: authenticate success, global_id 4687 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e13c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e10a15640 10 monclient: dump: 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:45.493 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:45.494 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:45.494 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:45.494 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.494 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.491+0000 7f6e13c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e080017d0 con 0x7f6e0c06af60 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 1 -- 172.21.15.96:0/3501197135 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6e08001c20 con 0x7f6e0c06af60 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e13c7b640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e13c7b640 4 set_mon_vals no callback set 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.495+0000 7f6e10a15640 10 monclient: dump: 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.496 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.497 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:45.497 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:45.497 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:45.497 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:45.497 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 10 monclient: shutdown 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3501197135 >> v1:172.21.15.203:6789/0 conn(0x7f6e0c06af60 legacy=0x7f6e0c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3501197135 shutdown_connections 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3501197135 >> 172.21.15.96:0/3501197135 conn(0x7f6e0c0b4a80 msgr2=0x7f6e0c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3501197135 shutdown_connections 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.496+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3501197135 wait complete. 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 10 monclient: build_initial_monmap 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 10 monclient: monmap: 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:45.499 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 librados: starting msgr at 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 librados: starting objecter 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 Processor -- start 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 -- start start 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 librados: setting wanted keys 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 1 librados: calling monclient init 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.497+0000 7f6e13c7b640 10 monclient: init 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:45.500 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e0c06ab80 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e0c0b00b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e0c143ed0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c143220 con 0x7f6e0c06ab80 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c1451f0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c1d4e50 con 0x7f6e0c0b00b0 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:45.501 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.498+0000 7f6e13c7b640 10 monclient(hunting): authenticate will time out at 5943.780212s 2024-03-25T17:39:45.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.510+0000 7f6e11a17640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6e0c06ab80 0x7f6e0c137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59674/0 (socket says 172.21.15.96:59674) 2024-03-25T17:39:45.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.510+0000 7f6e11a17640 1 -- 172.21.15.96:0/3986040702 learned_addr learned my addr 172.21.15.96:0/3986040702 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:45.513 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.513+0000 7f6e027fc640 10 client.?.objecter ms_handle_connect 0x7f6e0c06ab80 2024-03-25T17:39:45.514 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.513+0000 7f6e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.518 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.517+0000 7f6e027fc640 10 client.?.objecter ms_handle_connect 0x7f6e0c143ed0 2024-03-25T17:39:45.518 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.517+0000 7f6e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 client.?.objecter ms_handle_connect 0x7f6e0c0b00b0 2024-03-25T17:39:45.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:45.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159225202 0 0) 0x7f6e0c1451f0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.523 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T17:39:45.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.522+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0004eb0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.530+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559109415 0 0) 0x7f6df0004eb0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.530+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0c1451f0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6dfc001770 con 0x7f6e0c143ed0 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:45.531 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): dump: 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123928436 0 0) 0x7f6e0c143220 con 0x7f6e0c06ab80 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:45.532 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:45.533 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T17:39:45.533 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.531+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0004eb0 con 0x7f6e0c06ab80 2024-03-25T17:39:45.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 819550241 0 0) 0x7f6e0c1451f0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 >> v1:172.21.15.96:6790/0 conn(0x7f6e0c0b00b0 legacy=0x7f6e0c1d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 >> v1:172.21.15.96:6789/0 conn(0x7f6e0c06ab80 legacy=0x7f6e0c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 1 monclient: found mon.b 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c142e60 con 0x7f6e0c143ed0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 5 monclient: authenticate success, global_id 4690 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 10 monclient: _renew_subs 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:39:45.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c1437d0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 10 monclient: _renew_subs 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c1382e0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.534+0000 7f6e13c7b640 1 librados: init done 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.535+0000 7f6e13c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.535+0000 7f6e13c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.535+0000 7f6e13c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:39:45.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.535+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c143050 con 0x7f6e0c143ed0 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dfc001ca0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f6dfc0037b0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e027fc640 10 monclient: dump: 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:45.813 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.812+0000 7f6e03fff640 4 set_mon_vals no callback set 2024-03-25T17:39:45.814 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.813+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f6dfc0039d0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.813+0000 7f6e11a17640 10 monclient: get_auth_request con 0x7f6df006dcb0 auth_method 0 2024-03-25T17:39:45.815 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.814+0000 7f6e027fc640 10 client.4690.objecter ms_handle_connect 0x7f6df006dcb0 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f6dfc0534f0 con 0x7f6e0c143ed0 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 10 client.4690.objecter ms_dispatch 0x7f6e0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6dfc04f000 con 0x7f6e0c143ed0 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:45.818 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:45.817+0000 7f6e027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:46.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.070+0000 7f6e13c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:46.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.070+0000 7f6e13c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:46.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.070+0000 7f6e13c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:39:46.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.070+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e0c06fb80 con 0x7f6e0c143ed0 2024-03-25T17:39:46.094 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.094+0000 7f6e027fc640 1 -- 172.21.15.96:0/3986040702 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f6dfc050f30 con 0x7f6e0c143ed0 2024-03-25T17:39:46.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.094+0000 7f6e027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:46.095 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.094+0000 7f6e027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:46.095 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:46.095 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 10 librados: watch_flush enter 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 10 librados: watch_flush exit 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 10 client.4690.objecter successfully canceled tick 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 >> v1:172.21.15.203:6800/3611964061 conn(0x7f6df006dcb0 legacy=0x7f6df0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 10 monclient: shutdown 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 >> v1:172.21.15.203:6789/0 conn(0x7f6e0c143ed0 legacy=0x7f6e0c1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 shutdown_connections 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 >> 172.21.15.96:0/3986040702 conn(0x7f6e0c0b4a80 msgr2=0x7f6e0c070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 shutdown_connections 2024-03-25T17:39:46.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.096+0000 7f6e13c7b640 1 -- 172.21.15.96:0/3986040702 wait complete. 2024-03-25T17:39:46.099 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:46.097+0000 7f6e13c7b640 1 librados: shutdown 2024-03-25T17:39:46.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:52.140 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:16.366 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.149 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:16.376 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.183 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:16.410 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.197 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:16.412 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.203 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:16.430 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.207 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:16.434 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.218 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:16.434 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.245 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:16.472 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.250 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:16.465 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:52.916 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.035+0000 7fba3ffc6640 10 monclient: get_monmap_and_config 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.035+0000 7fba3ffc6640 10 monclient: build_initial_monmap 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.035+0000 7fba3ffc6640 10 monclient: monmap: 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:39:53.036762+0000 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:39:53.036762+0000 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.036 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:53.037 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:39:53.037 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:39:53.037 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.036+0000 7fba3ffc6640 1 Processor -- start 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.036+0000 7fba3ffc6640 1 -- start start 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.036+0000 7fba3ffc6640 10 monclient: init 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.036+0000 7fba3ffc6640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.036+0000 7fba3ffc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-a con 0x7fba380a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-c con 0x7fba3813f710 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:53.038 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-b con 0x7fba381430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38142520 con 0x7fba380a98f0 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380b28a0 con 0x7fba3813f710 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380b2a30 con 0x7fba381430c0 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:53.039 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.037+0000 7fba3ffc6640 10 monclient(hunting): authenticate will time out at 5951.319109s 2024-03-25T17:39:53.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.043+0000 7fba3dd62640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fba380a98f0 0x7fba380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36220/0 (socket says 172.21.15.96:36220) 2024-03-25T17:39:53.044 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.043+0000 7fba3dd62640 1 -- 172.21.15.96:0/2870536479 learned_addr learned my addr 172.21.15.96:0/2870536479 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747705453 0 0) 0x7fba38142520 con 0x7fba380a98f0 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 10 monclient(hunting): my global_id is 4716 2024-03-25T17:39:53.060 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.059+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24003520 con 0x7fba380a98f0 2024-03-25T17:39:53.064 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128653339 0 0) 0x7fba380b28a0 con 0x7fba3813f710 2024-03-25T17:39:53.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 10 monclient(hunting): my global_id is 4693 2024-03-25T17:39:53.065 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.064+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38142520 con 0x7fba3813f710 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647988617 0 0) 0x7fba380b2a30 con 0x7fba381430c0 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 10 monclient(hunting): my global_id is 4697 2024-03-25T17:39:53.069 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.068+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba380b28a0 con 0x7fba381430c0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580544219 0 0) 0x7fba24003520 con 0x7fba380a98f0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 >> v1:172.21.15.96:6790/0 conn(0x7fba381430c0 legacy=0x7fba3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 >> v1:172.21.15.203:6789/0 conn(0x7fba3813f710 legacy=0x7fba381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: _finish_hunting 0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 monclient: found mon.noname-a 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380b44c0 con 0x7fba380a98f0 2024-03-25T17:39:53.070 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: _finish_auth 0 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: _check_auth_tickets 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba34001690 con 0x7fba380a98f0 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3cd60640 10 monclient: dump: 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.071 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3ffc6640 5 monclient: authenticate success, global_id 4716 2024-03-25T17:39:53.072 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.069+0000 7fba3ffc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba34001870 con 0x7fba380a98f0 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 1 -- 172.21.15.96:0/2870536479 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba34001c80 con 0x7fba380a98f0 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3ffc6640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3ffc6640 4 set_mon_vals no callback set 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.081+0000 7fba3cd60640 10 monclient: dump: 2024-03-25T17:39:53.082 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.083 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.084 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 10 monclient: shutdown 2024-03-25T17:39:53.084 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 1 -- 172.21.15.96:0/2870536479 >> v1:172.21.15.96:6789/0 conn(0x7fba380a98f0 legacy=0x7fba380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 1 -- 172.21.15.96:0/2870536479 shutdown_connections 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 1 -- 172.21.15.96:0/2870536479 >> 172.21.15.96:0/2870536479 conn(0x7fba380b7ea0 msgr2=0x7fba380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 1 -- 172.21.15.96:0/2870536479 shutdown_connections 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.082+0000 7fba3ffc6640 1 -- 172.21.15.96:0/2870536479 wait complete. 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 10 monclient: build_initial_monmap 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 10 monclient: monmap: 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:39:53.085 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 librados: starting msgr at 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 librados: starting objecter 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 Processor -- start 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 -- start start 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 librados: setting wanted keys 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 1 librados: calling monclient init 2024-03-25T17:39:53.086 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.083+0000 7fba3ffc6640 10 monclient: init 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.084+0000 7fba3ffc6640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.084+0000 7fba3ffc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.084+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-b con 0x7fba380a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.084+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-a con 0x7fba3813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.084+0000 7fba3ffc6640 10 monclient(hunting): picked mon.noname-c con 0x7fba381430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.085+0000 7fba3ffc6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3813c840 con 0x7fba3813f710 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.085+0000 7fba3ffc6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba381d41e0 con 0x7fba380a98f0 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.085+0000 7fba3ffc6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba381d4370 con 0x7fba381430c0 2024-03-25T17:39:53.087 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.085+0000 7fba3ffc6640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:53.088 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.085+0000 7fba3ffc6640 10 monclient(hunting): authenticate will time out at 5951.367036s 2024-03-25T17:39:53.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.096+0000 7fba3e563640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7fba381430c0 0x7fba381db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:49854/0 (socket says 172.21.15.96:49854) 2024-03-25T17:39:53.096 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.096+0000 7fba3e563640 1 -- 172.21.15.96:0/4181678301 learned_addr learned my addr 172.21.15.96:0/4181678301 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:39:53.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.097+0000 7fba2e7fc640 10 client.?.objecter ms_handle_connect 0x7fba381430c0 2024-03-25T17:39:53.098 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.097+0000 7fba2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.106+0000 7fba2e7fc640 10 client.?.objecter ms_handle_connect 0x7fba380a98f0 2024-03-25T17:39:53.107 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.106+0000 7fba2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.113 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499566554 0 0) 0x7fba381d4370 con 0x7fba381430c0 2024-03-25T17:39:53.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T17:39:53.114 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.113+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c004eb0 con 0x7fba381430c0 2024-03-25T17:39:53.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.115+0000 7fba2e7fc640 10 client.?.objecter ms_handle_connect 0x7fba3813f710 2024-03-25T17:39:53.116 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.115+0000 7fba2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:53.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077818314 0 0) 0x7fba3813c840 con 0x7fba3813f710 2024-03-25T17:39:53.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.188 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T17:39:53.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.187+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba381d4370 con 0x7fba3813f710 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202376897 0 0) 0x7fba381d41e0 con 0x7fba380a98f0 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 10 monclient(hunting): my global_id is 4696 2024-03-25T17:39:53.198 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.197+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3813c840 con 0x7fba380a98f0 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682547055 0 0) 0x7fba1c004eb0 con 0x7fba381430c0 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba381d41e0 con 0x7fba381430c0 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba30001c70 con 0x7fba381430c0 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 10 monclient(hunting): dump: 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.203 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.204 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.204 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.204 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.204 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.204 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.202+0000 7fba2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279150266 0 0) 0x7fba3813c840 con 0x7fba380a98f0 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba1c004eb0 con 0x7fba380a98f0 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba34001b80 con 0x7fba380a98f0 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.218+0000 7fba2e7fc640 10 monclient(hunting): dump: 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.219 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.220 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.220 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.220 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.220 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873170885 0 0) 0x7fba381d4370 con 0x7fba3813f710 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba3813c840 con 0x7fba3813f710 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba28001930 con 0x7fba3813f710 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.224+0000 7fba2e7fc640 10 monclient(hunting): dump: 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.225 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.226 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.226 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.226 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.226 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271539174 0 0) 0x7fba381d41e0 con 0x7fba381430c0 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 >> v1:172.21.15.203:6789/0 conn(0x7fba380a98f0 legacy=0x7fba381d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 >> v1:172.21.15.96:6789/0 conn(0x7fba3813f710 legacy=0x7fba381dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 1 monclient: found mon.c 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38135ba0 con 0x7fba381430c0 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba3ffc6640 5 monclient: authenticate success, global_id 4700 2024-03-25T17:39:53.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba3ffc6640 10 monclient: _renew_subs 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba3ffc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.318+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba38135d90 con 0x7fba381430c0 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.319+0000 7fba3ffc6640 10 client.4700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.319+0000 7fba3ffc6640 10 monclient: _renew_subs 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.319+0000 7fba3ffc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.319+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba381d5150 con 0x7fba381430c0 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.319+0000 7fba3ffc6640 1 librados: init done 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.320+0000 7fba3ffc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.320+0000 7fba3ffc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.320+0000 7fba3ffc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:53.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.320+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38136e10 con 0x7fba381430c0 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba30003990 con 0x7fba381430c0 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fba30003da0 con 0x7fba381430c0 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2e7fc640 10 monclient: dump: 2024-03-25T17:39:53.627 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:39:53.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.626+0000 7fba2ffff640 4 set_mon_vals no callback set 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.627+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fba30003fc0 con 0x7fba381430c0 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.628+0000 7fba3dd62640 10 monclient: get_auth_request con 0x7fba1c06e0f0 auth_method 0 2024-03-25T17:39:53.629 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.628+0000 7fba2e7fc640 10 client.4700.objecter ms_handle_connect 0x7fba1c06e0f0 2024-03-25T17:39:53.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7fba30054b10 con 0x7fba381430c0 2024-03-25T17:39:53.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 10 client.4700.objecter ms_dispatch 0x7fba38000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:39:53.636 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 3 client.4700.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:39:53.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 3 client.4700.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:39:53.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 20 client.4700.objecter dump_active .. 0 homeless 2024-03-25T17:39:53.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba30050620 con 0x7fba381430c0 2024-03-25T17:39:53.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:53.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.636+0000 7fba2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:53.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.894+0000 7fba3ffc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.894+0000 7fba3ffc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.894+0000 7fba3ffc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:39:53.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.894+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba380627b0 con 0x7fba381430c0 2024-03-25T17:39:53.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.939+0000 7fba2e7fc640 1 -- 172.21.15.96:0/4181678301 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7fba30052550 con 0x7fba381430c0 2024-03-25T17:39:53.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.939+0000 7fba2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:53.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.939+0000 7fba2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:53.940 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:39:53.940 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 10 librados: watch_flush enter 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 10 librados: watch_flush exit 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 20 client.4700.objecter shutdown clearing up homeless session... 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 10 client.4700.objecter successfully canceled tick 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 >> v1:172.21.15.203:6800/3611964061 conn(0x7fba1c06e0f0 legacy=0x7fba1c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 10 monclient: shutdown 2024-03-25T17:39:53.943 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 >> v1:172.21.15.96:6790/0 conn(0x7fba381430c0 legacy=0x7fba381db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 shutdown_connections 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 >> 172.21.15.96:0/4181678301 conn(0x7fba380b7ea0 msgr2=0x7fba38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 shutdown_connections 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 -- 172.21.15.96:0/4181678301 wait complete. 2024-03-25T17:39:53.944 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:39:53.941+0000 7fba3ffc6640 1 librados: shutdown 2024-03-25T17:39:53.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:59.423 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:23.649 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.434 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:23.661 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.439 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:23.655 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.444 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:23.670 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.444 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:23.659 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.447 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:23.674 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.468 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:23.695 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.507 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:23.722 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:59.532 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:23.759 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:00.760 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:00.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.873+0000 7fb59b5c6640 10 monclient: get_monmap_and_config 2024-03-25T17:40:00.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.873+0000 7fb59b5c6640 10 monclient: build_initial_monmap 2024-03-25T17:40:00.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.873+0000 7fb59b5c6640 10 monclient: monmap: 2024-03-25T17:40:00.873 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:00.873 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:40:00.874137+0000 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:40:00.874137+0000 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:40:00.874 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:00.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.874+0000 7fb59b5c6640 1 Processor -- start 2024-03-25T17:40:00.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.874+0000 7fb59b5c6640 1 -- start start 2024-03-25T17:40:00.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.874+0000 7fb59b5c6640 10 monclient: init 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5940b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5940b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb59414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940af940 con 0x7fb5940b1730 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940b0a70 con 0x7fb59414be90 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940b0d50 con 0x7fb5940b4d60 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:00.876 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.875+0000 7fb59b5c6640 10 monclient(hunting): authenticate will time out at 5959.156393s 2024-03-25T17:40:00.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.885+0000 7fb599b63640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb59414be90 0x7fb59414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:54032/0 (socket says 172.21.15.96:54032) 2024-03-25T17:40:00.884 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.885+0000 7fb599b63640 1 -- 172.21.15.96:0/984493093 learned_addr learned my addr 172.21.15.96:0/984493093 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348511780 0 0) 0x7fb5940b0a70 con 0x7fb59414be90 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 10 monclient(hunting): my global_id is 4699 2024-03-25T17:40:00.904 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.904+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580003520 con 0x7fb59414be90 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421635077 0 0) 0x7fb5940b0d50 con 0x7fb5940b4d60 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 10 monclient(hunting): my global_id is 4703 2024-03-25T17:40:00.910 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.910+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5940b0a70 con 0x7fb5940b4d60 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790172238 0 0) 0x7fb5940af940 con 0x7fb5940b1730 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 10 monclient(hunting): my global_id is 4722 2024-03-25T17:40:00.918 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.918+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5940b0d50 con 0x7fb5940b1730 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086634960 0 0) 0x7fb580003520 con 0x7fb59414be90 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 >> v1:172.21.15.96:6790/0 conn(0x7fb5940b4d60 legacy=0x7fb59413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 >> v1:172.21.15.96:6789/0 conn(0x7fb5940b1730 legacy=0x7fb5940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 monclient: found mon.noname-c 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940b1030 con 0x7fb59414be90 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: _finish_auth 0 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:00.922 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb58c001a10 con 0x7fb59414be90 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb59b5c6640 5 monclient: authenticate success, global_id 4699 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb59b5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb58bfff640 10 monclient: dump: 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:00.923 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:00.924 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.922+0000 7fb59b5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb58c001b70 con 0x7fb59414be90 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 1 -- 172.21.15.96:0/984493093 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb58c001f80 con 0x7fb59414be90 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb58bfff640 10 monclient: dump: 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:00.936 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb59b5c6640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:00.937 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.936+0000 7fb59b5c6640 4 set_mon_vals no callback set 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 10 monclient: shutdown 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 1 -- 172.21.15.96:0/984493093 >> v1:172.21.15.203:6789/0 conn(0x7fb59414be90 legacy=0x7fb59414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 1 -- 172.21.15.96:0/984493093 shutdown_connections 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 1 -- 172.21.15.96:0/984493093 >> 172.21.15.96:0/984493093 conn(0x7fb5940617a0 msgr2=0x7fb594061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:00.939 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 1 -- 172.21.15.96:0/984493093 shutdown_connections 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.937+0000 7fb59b5c6640 1 -- 172.21.15.96:0/984493093 wait complete. 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.938+0000 7fb59b5c6640 10 monclient: build_initial_monmap 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.938+0000 7fb59b5c6640 10 monclient: monmap: 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 librados: starting msgr at 2024-03-25T17:40:00.940 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 librados: starting objecter 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 Processor -- start 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 -- start start 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 librados: setting wanted keys 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 librados: calling monclient init 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient: init 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5940b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5940b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb59414be90 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb594143900 con 0x7fb5940b1730 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb594137de0 con 0x7fb59414be90 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb594138050 con 0x7fb5940b4d60 2024-03-25T17:40:00.941 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:00.942 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.939+0000 7fb59b5c6640 10 monclient(hunting): authenticate will time out at 5959.220812s 2024-03-25T17:40:00.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.955+0000 7fb599362640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb5940b1730 0x7fb5941db400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33438/0 (socket says 172.21.15.96:33438) 2024-03-25T17:40:00.955 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.955+0000 7fb599362640 1 -- 172.21.15.96:0/861076477 learned_addr learned my addr 172.21.15.96:0/861076477 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:00.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.956+0000 7fb589ffb640 10 client.?.objecter ms_handle_connect 0x7fb5940b1730 2024-03-25T17:40:00.956 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.956+0000 7fb589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.961+0000 7fb589ffb640 10 client.?.objecter ms_handle_connect 0x7fb59414be90 2024-03-25T17:40:00.960 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.961+0000 7fb589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.964 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.965+0000 7fb589ffb640 10 client.?.objecter ms_handle_connect 0x7fb5940b4d60 2024-03-25T17:40:00.965 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.965+0000 7fb589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535158612 0 0) 0x7fb594143900 con 0x7fb5940b1730 2024-03-25T17:40:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.970 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 10 monclient(hunting): my global_id is 4725 2024-03-25T17:40:00.971 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.971+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb5940b1730 2024-03-25T17:40:00.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191723341 0 0) 0x7fb594137de0 con 0x7fb59414be90 2024-03-25T17:40:00.974 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 10 monclient(hunting): my global_id is 4702 2024-03-25T17:40:00.975 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.975+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb594143900 con 0x7fb59414be90 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502086041 0 0) 0x7fb594138050 con 0x7fb5940b4d60 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 10 monclient(hunting): my global_id is 4706 2024-03-25T17:40:00.982 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.982+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb594137de0 con 0x7fb5940b4d60 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1045758055 0 0) 0x7fb594143900 con 0x7fb59414be90 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb594138050 con 0x7fb59414be90 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb58c001e80 con 0x7fb59414be90 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 10 monclient(hunting): dump: 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:00.989 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:00.990 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:00.990 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:00.990 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:00.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:00.989+0000 7fb589ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345036367 0 0) 0x7fb578004eb0 con 0x7fb5940b1730 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb594143900 con 0x7fb5940b1730 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb590001a30 con 0x7fb5940b1730 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.010+0000 7fb589ffb640 10 monclient(hunting): dump: 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:01.010 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:01.011 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:01.011 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:01.011 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:01.011 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44128552 0 0) 0x7fb594137de0 con 0x7fb5940b4d60 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb5940b4d60 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb584001830 con 0x7fb5940b4d60 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.012+0000 7fb589ffb640 10 monclient(hunting): dump: 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:01.012 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:01.013 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 895871175 0 0) 0x7fb594138050 con 0x7fb59414be90 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 >> v1:172.21.15.96:6790/0 conn(0x7fb5940b4d60 legacy=0x7fb594143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 >> v1:172.21.15.96:6789/0 conn(0x7fb5940b1730 legacy=0x7fb5941db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 1 monclient: found mon.b 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5941dbb10 con 0x7fb59414be90 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 10 monclient: _finish_auth 0 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:01.018 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb59b5c6640 5 monclient: authenticate success, global_id 4702 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb59b5c6640 10 monclient: _renew_subs 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb59b5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.017+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5941dbd60 con 0x7fb59414be90 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.018+0000 7fb59b5c6640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.018+0000 7fb59b5c6640 10 monclient: _renew_subs 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.018+0000 7fb59b5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.018+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941e38c0 con 0x7fb59414be90 2024-03-25T17:40:01.019 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.018+0000 7fb59b5c6640 1 librados: init done 2024-03-25T17:40:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.019+0000 7fb59b5c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.019+0000 7fb59b5c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.019+0000 7fb59b5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:01.021 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.019+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5941392d0 con 0x7fb59414be90 2024-03-25T17:40:01.320 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb58c001fe0 con 0x7fb59414be90 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7fb58c004e10 con 0x7fb59414be90 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 10 monclient: dump: 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:01.321 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:01.322 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:01.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb58b7fe640 4 set_mon_vals no callback set 2024-03-25T17:40:01.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.321+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7fb58c01ade0 con 0x7fb59414be90 2024-03-25T17:40:01.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.322+0000 7fb599362640 10 monclient: get_auth_request con 0x7fb57806e0b0 auth_method 0 2024-03-25T17:40:01.322 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.322+0000 7fb589ffb640 10 client.4702.objecter ms_handle_connect 0x7fb57806e0b0 2024-03-25T17:40:01.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.328+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7fb58c054850 con 0x7fb59414be90 2024-03-25T17:40:01.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.328+0000 7fb589ffb640 10 client.4702.objecter ms_dispatch 0x7fb594000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:40:01.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.328+0000 7fb589ffb640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:40:01.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.328+0000 7fb589ffb640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:40:01.327 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.328+0000 7fb589ffb640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T17:40:01.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.329+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb58c050360 con 0x7fb59414be90 2024-03-25T17:40:01.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.329+0000 7fb589ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:01.329 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.329+0000 7fb589ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:01.569 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.569+0000 7fb59b5c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.569+0000 7fb59b5c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.569+0000 7fb59b5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:01.570 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.569+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5940627b0 con 0x7fb59414be90 2024-03-25T17:40:01.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.669+0000 7fb589ffb640 1 -- 172.21.15.96:0/861076477 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7fb58c052290 con 0x7fb59414be90 2024-03-25T17:40:01.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.669+0000 7fb589ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:01.669 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.669+0000 7fb589ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:01.669 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:40:01.669 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:01.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 10 librados: watch_flush enter 2024-03-25T17:40:01.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 10 librados: watch_flush exit 2024-03-25T17:40:01.671 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 10 client.4702.objecter successfully canceled tick 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 >> v1:172.21.15.203:6800/3611964061 conn(0x7fb57806e0b0 legacy=0x7fb578070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 10 monclient: shutdown 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 >> v1:172.21.15.203:6789/0 conn(0x7fb59414be90 legacy=0x7fb5941444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 shutdown_connections 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 >> 172.21.15.96:0/861076477 conn(0x7fb5940617a0 msgr2=0x7fb59413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 shutdown_connections 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 -- 172.21.15.96:0/861076477 wait complete. 2024-03-25T17:40:01.672 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:01.671+0000 7fb59b5c6640 1 librados: shutdown 2024-03-25T17:40:01.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:08.490 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:08.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.600+0000 7f9ce8272640 10 monclient: get_monmap_and_config 2024-03-25T17:40:08.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.600+0000 7f9ce8272640 10 monclient: build_initial_monmap 2024-03-25T17:40:08.601 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.601+0000 7f9ce8272640 10 monclient: monmap: 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:40:08.602269+0000 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:40:08.602269+0000 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:40:08.602 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.601+0000 7f9ce8272640 1 Processor -- start 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.601+0000 7f9ce8272640 1 -- start start 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.601+0000 7f9ce8272640 10 monclient: init 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce00b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:08.603 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce00b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce014be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00af940 con 0x7f9ce00b1730 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00b0a70 con 0x7f9ce00b4d60 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00b0d50 con 0x7f9ce014be90 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:08.604 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.602+0000 7f9ce8272640 10 monclient(hunting): authenticate will time out at 5966.884712s 2024-03-25T17:40:08.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.607+0000 7f9ce600e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9ce00b4d60 0x7f9ce013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:52606/0 (socket says 172.21.15.96:52606) 2024-03-25T17:40:08.607 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.607+0000 7f9ce600e640 1 -- 172.21.15.96:0/527591016 learned_addr learned my addr 172.21.15.96:0/527591016 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830895197 0 0) 0x7f9ce00b0a70 con 0x7f9ce00b4d60 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 10 monclient(hunting): my global_id is 4705 2024-03-25T17:40:08.618 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.617+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ccc003520 con 0x7f9ce00b4d60 2024-03-25T17:40:08.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690015505 0 0) 0x7f9ccc003520 con 0x7f9ce00b4d60 2024-03-25T17:40:08.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 >> v1:172.21.15.96:6790/0 conn(0x7f9ce014be90 legacy=0x7f9ce014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 >> v1:172.21.15.96:6789/0 conn(0x7f9ce00b1730 legacy=0x7f9ce00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.626 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: _finish_hunting 0 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 monclient: found mon.noname-c 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce00b1030 con 0x7f9ce00b4d60 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: _finish_auth 0 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: _check_auth_tickets 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9cdc001690 con 0x7f9ce00b4d60 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce500c640 10 monclient: dump: 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:08.627 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce8272640 5 monclient: authenticate success, global_id 4705 2024-03-25T17:40:08.628 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.625+0000 7f9ce8272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:08.634 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cdc0017f0 con 0x7f9ce00b4d60 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 1 -- 172.21.15.96:0/527591016 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9cdc001c40 con 0x7f9ce00b4d60 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce8272640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce8272640 4 set_mon_vals no callback set 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce500c640 10 monclient: dump: 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:08.635 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:08.636 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:08.636 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:08.636 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.634+0000 7f9ce8272640 10 monclient: shutdown 2024-03-25T17:40:08.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:08.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 1 -- 172.21.15.96:0/527591016 >> v1:172.21.15.203:6789/0 conn(0x7f9ce00b4d60 legacy=0x7f9ce013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 1 -- 172.21.15.96:0/527591016 shutdown_connections 2024-03-25T17:40:08.637 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 1 -- 172.21.15.96:0/527591016 >> 172.21.15.96:0/527591016 conn(0x7f9ce00617a0 msgr2=0x7f9ce0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 1 -- 172.21.15.96:0/527591016 shutdown_connections 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.635+0000 7f9ce8272640 1 -- 172.21.15.96:0/527591016 wait complete. 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 10 monclient: build_initial_monmap 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 10 monclient: monmap: 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.638 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 librados: starting msgr at 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 librados: starting objecter 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 Processor -- start 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 -- start start 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 librados: setting wanted keys 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 1 librados: calling monclient init 2024-03-25T17:40:08.639 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.636+0000 7f9ce8272640 10 monclient: init 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce00b1730 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce00b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce014be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce01d7940 con 0x7f9ce00b1730 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce01382f0 con 0x7f9ce00b4d60 2024-03-25T17:40:08.640 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce0137320 con 0x7f9ce014be90 2024-03-25T17:40:08.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:08.641 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.637+0000 7f9ce8272640 10 monclient(hunting): authenticate will time out at 5966.919681s 2024-03-25T17:40:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.648+0000 7f9ce680f640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f9ce014be90 0x7f9ce012fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:47840/0 (socket says 172.21.15.96:47840) 2024-03-25T17:40:08.649 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.648+0000 7f9ce680f640 1 -- 172.21.15.96:0/687680933 learned_addr learned my addr 172.21.15.96:0/687680933 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.650+0000 7f9cd6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ce00b1730 2024-03-25T17:40:08.650 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.650+0000 7f9cd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.652+0000 7f9cd6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ce014be90 2024-03-25T17:40:08.652 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.652+0000 7f9cd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279922833 0 0) 0x7f9ce01d7940 con 0x7f9ce00b1730 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 10 monclient(hunting): my global_id is 4731 2024-03-25T17:40:08.654 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.653+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc4004eb0 con 0x7f9ce00b1730 2024-03-25T17:40:08.662 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139816958 0 0) 0x7f9ce0137320 con 0x7f9ce014be90 2024-03-25T17:40:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T17:40:08.663 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.662+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ce01d7940 con 0x7f9ce014be90 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009387380 0 0) 0x7f9cc4004eb0 con 0x7f9ce00b1730 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ce0137320 con 0x7f9ce00b1730 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9cdc001b00 con 0x7f9ce00b1730 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 10 monclient(hunting): dump: 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:08.664 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.665 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.663+0000 7f9cd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:08.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.667+0000 7f9cd6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ce00b4d60 2024-03-25T17:40:08.668 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.667+0000 7f9cd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610585102 0 0) 0x7f9ce0137320 con 0x7f9ce00b1730 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 >> v1:172.21.15.96:6790/0 conn(0x7f9ce014be90 legacy=0x7f9ce012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 >> v1:172.21.15.203:6789/0 conn(0x7f9ce00b4d60 legacy=0x7f9ce01d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 1 monclient: found mon.a 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce01d8530 con 0x7f9ce00b1730 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:40:08.683 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9cd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 5 monclient: authenticate success, global_id 4731 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 10 monclient: _renew_subs 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce0130570 con 0x7f9ce00b1730 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 10 monclient: _renew_subs 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce0143ae0 con 0x7f9ce00b1730 2024-03-25T17:40:08.684 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.682+0000 7f9ce8272640 1 librados: init done 2024-03-25T17:40:08.685 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.683+0000 7f9ce8272640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.685 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.683+0000 7f9ce8272640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.685 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.683+0000 7f9ce8272640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:40:08.685 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.683+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce01d77a0 con 0x7f9ce00b1730 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cdc001c60 con 0x7f9ce00b1730 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9cdc004bf0 con 0x7f9ce00b1730 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9cd6ffd640 10 monclient: dump: 2024-03-25T17:40:08.990 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:08.991 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.989+0000 7f9ce480b640 4 set_mon_vals no callback set 2024-03-25T17:40:08.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.990+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f9cdc01abc0 con 0x7f9ce00b1730 2024-03-25T17:40:08.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.990+0000 7f9ce580d640 10 monclient: get_auth_request con 0x7f9cc406dd40 auth_method 0 2024-03-25T17:40:08.993 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.991+0000 7f9cd6ffd640 10 client.4731.objecter ms_handle_connect 0x7f9cc406dd40 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f9cdc0546c0 con 0x7f9ce00b1730 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 10 client.4731.objecter ms_dispatch 0x7f9ce0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9cdc0501d0 con 0x7f9ce00b1730 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:08.998 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:08.997+0000 7f9cd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:09.222 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.221+0000 7f9ce8272640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.222+0000 7f9ce8272640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.222+0000 7f9ce8272640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.96:6789/0 2024-03-25T17:40:09.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.222+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 --> v1:172.21.15.96:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce01d86c0 con 0x7f9ce00b1730 2024-03-25T17:40:09.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.252+0000 7f9cd6ffd640 1 -- 172.21.15.96:0/687680933 <== mon.0 v1:172.21.15.96:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f9ce01d86c0 con 0x7f9ce00b1730 2024-03-25T17:40:09.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.252+0000 7f9cd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:09.252 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.252+0000 7f9cd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:09.253 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:40:09.253 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 10 librados: watch_flush enter 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 10 librados: watch_flush exit 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 10 client.4731.objecter successfully canceled tick 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 >> v1:172.21.15.203:6800/3611964061 conn(0x7f9cc406dd40 legacy=0x7f9cc4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 10 monclient: shutdown 2024-03-25T17:40:09.255 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 >> v1:172.21.15.96:6789/0 conn(0x7f9ce00b1730 legacy=0x7f9ce0136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 shutdown_connections 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 >> 172.21.15.96:0/687680933 conn(0x7f9ce00617a0 msgr2=0x7f9ce00b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 shutdown_connections 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 -- 172.21.15.96:0/687680933 wait complete. 2024-03-25T17:40:09.256 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:09.254+0000 7f9ce8272640 1 librados: shutdown 2024-03-25T17:40:09.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:12.165 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:36.391 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.215 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:36.442 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.236 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:36.463 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.257 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:36.484 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.265 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:36.492 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.278 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:36.493 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.287 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:36.502 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.308 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:36.534 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:12.331 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:36.546 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.073 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.186+0000 7f9766872640 10 monclient: get_monmap_and_config 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.186+0000 7f9766872640 10 monclient: build_initial_monmap 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.186+0000 7f9766872640 10 monclient: monmap: 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:40:16.187575+0000 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:40:16.187575+0000 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:16.187 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:40:16.188 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:40:16.188 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.187+0000 7f9766872640 1 Processor -- start 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.187+0000 7f9766872640 1 -- start start 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.187+0000 7f9766872640 10 monclient: init 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient(hunting): picked mon.noname-b con 0x7f97600705b0 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient(hunting): picked mon.noname-a con 0x7f97601402d0 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient(hunting): picked mon.noname-c con 0x7f9760149ae0 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600718c0 con 0x7f97601402d0 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600714c0 con 0x7f9760149ae0 2024-03-25T17:40:16.189 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9760072b50 con 0x7f97600705b0 2024-03-25T17:40:16.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:16.190 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.188+0000 7f9766872640 10 monclient(hunting): authenticate will time out at 5974.470266s 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.195+0000 7f975f7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f97601402d0 0x7f9760140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46608/0 (socket says 172.21.15.96:46608) 2024-03-25T17:40:16.196 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.195+0000 7f975f7fe640 1 -- 172.21.15.96:0/3010571343 learned_addr learned my addr 172.21.15.96:0/3010571343 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393657444 0 0) 0x7f97600718c0 con 0x7f97601402d0 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 10 monclient(hunting): my global_id is 4734 2024-03-25T17:40:16.211 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.210+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748003520 con 0x7f97601402d0 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264636752 0 0) 0x7f97600714c0 con 0x7f9760149ae0 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 10 monclient(hunting): my global_id is 4711 2024-03-25T17:40:16.217 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.216+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97600718c0 con 0x7f9760149ae0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088116273 0 0) 0x7f9748003520 con 0x7f97601402d0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 >> v1:172.21.15.96:6790/0 conn(0x7f97600705b0 legacy=0x7f976013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 >> v1:172.21.15.203:6789/0 conn(0x7f9760149ae0 legacy=0x7f976014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 10 monclient: _finish_hunting 0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 monclient: found mon.noname-a 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976013dd10 con 0x7f97601402d0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 10 monclient: _finish_auth 0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 10 monclient: _check_auth_tickets 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9754001690 con 0x7f97601402d0 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f9766872640 5 monclient: authenticate success, global_id 4734 2024-03-25T17:40:16.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f9766872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.221+0000 7f975effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.222+0000 7f975effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.222+0000 7f975effd640 10 monclient: dump: 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.222+0000 7f9766872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97540017f0 con 0x7f97601402d0 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 1 -- 172.21.15.96:0/3010571343 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9754001c00 con 0x7f97601402d0 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f975effd640 10 monclient: dump: 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:16.245 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f9766872640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:16.246 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.244+0000 7f9766872640 4 set_mon_vals no callback set 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 10 monclient: shutdown 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 1 -- 172.21.15.96:0/3010571343 >> v1:172.21.15.96:6789/0 conn(0x7f97601402d0 legacy=0x7f9760140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 1 -- 172.21.15.96:0/3010571343 shutdown_connections 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 1 -- 172.21.15.96:0/3010571343 >> 172.21.15.96:0/3010571343 conn(0x7f97600b5d10 msgr2=0x7f97600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 1 -- 172.21.15.96:0/3010571343 shutdown_connections 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.245+0000 7f9766872640 1 -- 172.21.15.96:0/3010571343 wait complete. 2024-03-25T17:40:16.248 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.246+0000 7f9766872640 10 monclient: build_initial_monmap 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.246+0000 7f9766872640 10 monclient: monmap: 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.246+0000 7f9766872640 1 librados: starting msgr at 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.246+0000 7f9766872640 1 librados: starting objecter 2024-03-25T17:40:16.249 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.246+0000 7f9766872640 1 Processor -- start 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 -- start start 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 librados: setting wanted keys 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 librados: calling monclient init 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient: init 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient(hunting): picked mon.noname-a con 0x7f97600705b0 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient(hunting): picked mon.noname-b con 0x7f97601402d0 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient(hunting): picked mon.noname-c con 0x7f9760149ae0 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97601418b0 con 0x7f97600705b0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9760141b50 con 0x7f97601402d0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9760143c30 con 0x7f9760149ae0 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:16.250 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.247+0000 7f9766872640 10 monclient(hunting): authenticate will time out at 5974.529803s 2024-03-25T17:40:16.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.257+0000 7f975f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f97601402d0 0x7f97601d55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:45394/0 (socket says 172.21.15.96:45394) 2024-03-25T17:40:16.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.257+0000 7f975f7fe640 1 -- 172.21.15.96:0/230721883 learned_addr learned my addr 172.21.15.96:0/230721883 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:16.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.258+0000 7f975cff9640 10 client.?.objecter ms_handle_connect 0x7f97601402d0 2024-03-25T17:40:16.258 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.258+0000 7f975cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788699670 0 0) 0x7f9760141b50 con 0x7f97601402d0 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 10 monclient(hunting): my global_id is 4714 2024-03-25T17:40:16.265 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.264+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c004eb0 con 0x7f97601402d0 2024-03-25T17:40:16.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.268+0000 7f975cff9640 10 client.?.objecter ms_handle_connect 0x7f97600705b0 2024-03-25T17:40:16.269 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.268+0000 7f975cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.273+0000 7f975cff9640 10 client.?.objecter ms_handle_connect 0x7f9760149ae0 2024-03-25T17:40:16.274 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.273+0000 7f975cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:16.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.275+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710826044 0 0) 0x7f974c004eb0 con 0x7f97601402d0 2024-03-25T17:40:16.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.275+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9760141b50 con 0x7f97601402d0 2024-03-25T17:40:16.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.275+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9754001bc0 con 0x7f97601402d0 2024-03-25T17:40:16.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.276+0000 7f975cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.276 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.276+0000 7f975cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.276+0000 7f975cff9640 10 monclient(hunting): dump: 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.277 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.276+0000 7f975cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040557521 0 0) 0x7f97601418b0 con 0x7f97600705b0 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 10 monclient(hunting): my global_id is 4737 2024-03-25T17:40:16.278 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.277+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c004eb0 con 0x7f97600705b0 2024-03-25T17:40:16.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 125734494 0 0) 0x7f9760141b50 con 0x7f97601402d0 2024-03-25T17:40:16.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 >> v1:172.21.15.96:6790/0 conn(0x7f9760149ae0 legacy=0x7f97601e0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 >> v1:172.21.15.96:6789/0 conn(0x7f97600705b0 legacy=0x7f9760133ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:40:16.283 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 1 monclient: found mon.b 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760141d90 con 0x7f97601402d0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 10 monclient: _finish_auth 0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f975cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 5 monclient: authenticate success, global_id 4714 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 10 monclient: _renew_subs 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9760144330 con 0x7f97601402d0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 10 monclient: _renew_subs 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9760142630 con 0x7f97601402d0 2024-03-25T17:40:16.284 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.282+0000 7f9766872640 1 librados: init done 2024-03-25T17:40:16.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.283+0000 7f9766872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.283+0000 7f9766872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.283+0000 7f9766872640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:16.285 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.283+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760142000 con 0x7f97601402d0 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97540011f0 con 0x7f97601402d0 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f9754004c70 con 0x7f97601402d0 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975cff9640 10 monclient: dump: 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:16.304 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.303+0000 7f975e7fc640 4 set_mon_vals no callback set 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.304+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f9754004e90 con 0x7f97601402d0 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.305+0000 7f975ffff640 10 monclient: get_auth_request con 0x7f974c06dd40 auth_method 0 2024-03-25T17:40:16.305 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.305+0000 7f975cff9640 10 client.4714.objecter ms_handle_connect 0x7f974c06dd40 2024-03-25T17:40:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.307+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f9754054960 con 0x7f97601402d0 2024-03-25T17:40:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.307+0000 7f975cff9640 10 client.4714.objecter ms_dispatch 0x7f9760000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:40:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.307+0000 7f975cff9640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:40:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.307+0000 7f975cff9640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:40:16.308 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.307+0000 7f975cff9640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T17:40:16.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.582+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97540502a0 con 0x7f97601402d0 2024-03-25T17:40:16.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.582+0000 7f975cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:16.583 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.582+0000 7f975cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:16.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.832+0000 7f9766872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.832+0000 7f9766872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.832+0000 7f9766872640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:16.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.832+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97600627b0 con 0x7f97601402d0 2024-03-25T17:40:16.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.856+0000 7f975cff9640 1 -- 172.21.15.96:0/230721883 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f97540523a0 con 0x7f97601402d0 2024-03-25T17:40:16.856 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.856+0000 7f975cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:16.857 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.856+0000 7f975cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:16.857 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:40:16.857 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:16.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 10 librados: watch_flush enter 2024-03-25T17:40:16.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 10 librados: watch_flush exit 2024-03-25T17:40:16.859 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 10 client.4714.objecter successfully canceled tick 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 >> v1:172.21.15.203:6800/3611964061 conn(0x7f974c06dd40 legacy=0x7f974c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 10 monclient: shutdown 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 >> v1:172.21.15.203:6789/0 conn(0x7f97601402d0 legacy=0x7f97601d55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 shutdown_connections 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 >> 172.21.15.96:0/230721883 conn(0x7f97600b5d10 msgr2=0x7f9760140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 shutdown_connections 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 -- 172.21.15.96:0/230721883 wait complete. 2024-03-25T17:40:16.860 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:16.858+0000 7f9766872640 1 librados: shutdown 2024-03-25T17:40:16.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:23.678 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.793+0000 7f1b2fe25640 10 monclient: get_monmap_and_config 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.793+0000 7f1b2fe25640 10 monclient: build_initial_monmap 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.793+0000 7f1b2fe25640 10 monclient: monmap: 2024-03-25T17:40:23.794 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:40:23.794543+0000 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:40:23.794543+0000 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:40:23.795 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.794+0000 7f1b2fe25640 1 Processor -- start 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.794+0000 7f1b2fe25640 1 -- start start 2024-03-25T17:40:23.796 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.794+0000 7f1b2fe25640 10 monclient: init 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-c con 0x7f1b280a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-a con 0x7f1b2813f710 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-b con 0x7f1b281430c0 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b28142520 con 0x7f1b2813f710 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b280b28a0 con 0x7f1b280a98f0 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b280b2a30 con 0x7f1b281430c0 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:23.797 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.795+0000 7f1b2fe25640 10 monclient(hunting): authenticate will time out at 5982.077302s 2024-03-25T17:40:23.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.802+0000 7f1b2e3c2640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f1b281430c0 0x7f1b2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:49504/0 (socket says 172.21.15.96:49504) 2024-03-25T17:40:23.803 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.802+0000 7f1b2e3c2640 1 -- 172.21.15.96:0/1209493789 learned_addr learned my addr 172.21.15.96:0/1209493789 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235811990 0 0) 0x7f1b280b2a30 con 0x7f1b281430c0 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 10 monclient(hunting): my global_id is 4721 2024-03-25T17:40:23.813 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.812+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14003520 con 0x7f1b281430c0 2024-03-25T17:40:23.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282432343 0 0) 0x7f1b280b28a0 con 0x7f1b280a98f0 2024-03-25T17:40:23.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.822 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 10 monclient(hunting): my global_id is 4717 2024-03-25T17:40:23.823 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.822+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b280b2a30 con 0x7f1b280a98f0 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3143732397 0 0) 0x7f1b14003520 con 0x7f1b281430c0 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 >> v1:172.21.15.203:6789/0 conn(0x7f1b280a98f0 legacy=0x7f1b280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 >> v1:172.21.15.96:6789/0 conn(0x7f1b2813f710 legacy=0x7f1b281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: _finish_hunting 0 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 monclient: found mon.noname-b 2024-03-25T17:40:23.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.96:6790/0 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b280b44c0 con 0x7f1b281430c0 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: _finish_auth 0 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: _check_auth_tickets 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1b20001a10 con 0x7f1b281430c0 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2cbbf640 10 monclient: dump: 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:23.828 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2fe25640 5 monclient: authenticate success, global_id 4721 2024-03-25T17:40:23.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.826+0000 7f1b2fe25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.2 v1:172.21.15.96:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b20001b70 con 0x7f1b281430c0 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 1 -- 172.21.15.96:0/1209493789 <== mon.2 v1:172.21.15.96:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1b20001f80 con 0x7f1b281430c0 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2cbbf640 10 monclient: dump: 2024-03-25T17:40:23.833 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2fe25640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:23.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.832+0000 7f1b2fe25640 4 set_mon_vals no callback set 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 10 monclient: shutdown 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 1 -- 172.21.15.96:0/1209493789 >> v1:172.21.15.96:6790/0 conn(0x7f1b281430c0 legacy=0x7f1b2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 1 -- 172.21.15.96:0/1209493789 shutdown_connections 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 1 -- 172.21.15.96:0/1209493789 >> 172.21.15.96:0/1209493789 conn(0x7f1b280b7ea0 msgr2=0x7f1b280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:23.837 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 1 -- 172.21.15.96:0/1209493789 shutdown_connections 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.833+0000 7f1b2fe25640 1 -- 172.21.15.96:0/1209493789 wait complete. 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 10 monclient: build_initial_monmap 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 10 monclient: monmap: 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:40:23.838 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 librados: starting msgr at 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 librados: starting objecter 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 Processor -- start 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 -- start start 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 librados: setting wanted keys 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 1 librados: calling monclient init 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.834+0000 7f1b2fe25640 10 monclient: init 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:23.839 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-a con 0x7f1b280a98f0 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2813f710 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient(hunting): picked mon.noname-b con 0x7f1b281430c0 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b28137ab0 con 0x7f1b280a98f0 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b28138bd0 con 0x7f1b281430c0 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b28138e40 con 0x7f1b2813f710 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:23.840 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.835+0000 7f1b2fe25640 10 monclient(hunting): authenticate will time out at 5982.117707s 2024-03-25T17:40:23.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.848+0000 7f1b2d3c0640 1 --1- >> v1:172.21.15.96:6790/0 conn(0x7f1b2813f710 0x7f1b281df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.96:6790/0 says I am v1:172.21.15.96:49532/0 (socket says 172.21.15.96:49532) 2024-03-25T17:40:23.849 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.848+0000 7f1b2d3c0640 1 -- 172.21.15.96:0/660066359 learned_addr learned my addr 172.21.15.96:0/660066359 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:23.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.849+0000 7f1b1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1b2813f710 2024-03-25T17:40:23.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.849+0000 7f1b1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.849+0000 7f1b1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1b280a98f0 2024-03-25T17:40:23.850 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.849+0000 7f1b1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.855 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.855+0000 7f1b1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1b281430c0 2024-03-25T17:40:23.855 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.855+0000 7f1b1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:23.862 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350827075 0 0) 0x7f1b28137ab0 con 0x7f1b280a98f0 2024-03-25T17:40:23.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 10 monclient(hunting): my global_id is 4743 2024-03-25T17:40:23.863 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.862+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c004eb0 con 0x7f1b280a98f0 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272598276 0 0) 0x7f1b28138e40 con 0x7f1b2813f710 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T17:40:23.867 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.866+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28137ab0 con 0x7f1b2813f710 2024-03-25T17:40:23.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920653969 0 0) 0x7f1b0c004eb0 con 0x7f1b280a98f0 2024-03-25T17:40:23.873 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28138e40 con 0x7f1b280a98f0 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1b24001930 con 0x7f1b280a98f0 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 10 monclient(hunting): dump: 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:23.874 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:23.875 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:23.875 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:23.875 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.875 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.872+0000 7f1b1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078511327 0 0) 0x7f1b28138bd0 con 0x7f1b281430c0 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 10 monclient(hunting): my global_id is 4720 2024-03-25T17:40:23.881 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.880+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c004eb0 con 0x7f1b281430c0 2024-03-25T17:40:23.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.882+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200892539 0 0) 0x7f1b28137ab0 con 0x7f1b2813f710 2024-03-25T17:40:23.883 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.882+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28138bd0 con 0x7f1b2813f710 2024-03-25T17:40:23.894 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1b18001a50 con 0x7f1b2813f710 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 10 monclient(hunting): dump: 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:23.895 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2832223663 0 0) 0x7f1b28138bd0 con 0x7f1b2813f710 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 >> v1:172.21.15.203:6789/0 conn(0x7f1b281430c0 legacy=0x7f1b281e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 >> v1:172.21.15.96:6789/0 conn(0x7f1b280a98f0 legacy=0x7f1b28137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 1 monclient: found mon.c 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.892+0000 7f1b1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b1e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28139120 con 0x7f1b2813f710 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b1e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 5 monclient: authenticate success, global_id 4724 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 10 monclient: _renew_subs 2024-03-25T17:40:23.896 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b28139400 con 0x7f1b2813f710 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 10 monclient: _renew_subs 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b280b7880 con 0x7f1b2813f710 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.893+0000 7f1b2fe25640 1 librados: init done 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.894+0000 7f1b2fe25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.894+0000 7f1b2fe25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.894+0000 7f1b2fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:40:23.897 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:23.894+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b28137810 con 0x7f1b2813f710 2024-03-25T17:40:24.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b18001bb0 con 0x7f1b2813f710 2024-03-25T17:40:24.223 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f1b18003a60 con 0x7f1b2813f710 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1e7fc640 10 monclient: dump: 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:24.224 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:24.225 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:24.225 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:24.225 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:24.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.222+0000 7f1b1ffff640 4 set_mon_vals no callback set 2024-03-25T17:40:24.225 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.223+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f1b18003c80 con 0x7f1b2813f710 2024-03-25T17:40:24.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.223+0000 7f1b2dbc1640 10 monclient: get_auth_request con 0x7f1b0c06df20 auth_method 0 2024-03-25T17:40:24.226 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.224+0000 7f1b1e7fc640 10 client.4724.objecter ms_handle_connect 0x7f1b0c06df20 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f1b18054850 con 0x7f1b2813f710 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 10 client.4724.objecter ms_dispatch 0x7f1b28000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T17:40:24.229 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b18050360 con 0x7f1b2813f710 2024-03-25T17:40:24.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:24.230 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.228+0000 7f1b1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:24.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.473+0000 7f1b2fe25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.473+0000 7f1b2fe25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.473+0000 7f1b2fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.96:6790/0 2024-03-25T17:40:24.474 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.473+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 --> v1:172.21.15.96:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b280627b0 con 0x7f1b2813f710 2024-03-25T17:40:24.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.516+0000 7f1b1e7fc640 1 -- 172.21.15.96:0/660066359 <== mon.2 v1:172.21.15.96:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f1b18052290 con 0x7f1b2813f710 2024-03-25T17:40:24.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.516+0000 7f1b1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:24.517 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.516+0000 7f1b1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:24.518 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:40:24.518 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:24.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 10 librados: watch_flush enter 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 10 librados: watch_flush exit 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 10 client.4724.objecter successfully canceled tick 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 >> v1:172.21.15.203:6800/3611964061 conn(0x7f1b0c06df20 legacy=0x7f1b0c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 10 monclient: shutdown 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.518+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 >> v1:172.21.15.96:6790/0 conn(0x7f1b2813f710 legacy=0x7f1b281df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.519+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 shutdown_connections 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.519+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 >> 172.21.15.96:0/660066359 conn(0x7f1b280b7ea0 msgr2=0x7f1b28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:24.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.519+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 shutdown_connections 2024-03-25T17:40:24.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.519+0000 7f1b2fe25640 1 -- 172.21.15.96:0/660066359 wait complete. 2024-03-25T17:40:24.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:24.519+0000 7f1b2fe25640 1 librados: shutdown 2024-03-25T17:40:24.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:31.338 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.452+0000 7f7d833cd640 10 monclient: get_monmap_and_config 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.452+0000 7f7d833cd640 10 monclient: build_initial_monmap 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.452+0000 7f7d833cd640 10 monclient: monmap: 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:40:31.454161+0000 2024-03-25T17:40:31.453 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:40:31.454161+0000 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-b 2024-03-25T17:40:31.454 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.453+0000 7f7d833cd640 1 Processor -- start 2024-03-25T17:40:31.455 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.453+0000 7f7d833cd640 1 -- start start 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.453+0000 7f7d833cd640 10 monclient: init 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c0b4d60 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c14be90 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.454+0000 7f7d833cd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0af940 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.455+0000 7f7d833cd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0b0a70 con 0x7f7d7c0b1730 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.455+0000 7f7d833cd640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c0b0d50 con 0x7f7d7c14be90 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.455+0000 7f7d833cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:31.456 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.455+0000 7f7d833cd640 10 monclient(hunting): authenticate will time out at 5989.737158s 2024-03-25T17:40:31.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.467+0000 7f7d81169640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7d7c0b4d60 0x7f7d7c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50072/0 (socket says 172.21.15.96:50072) 2024-03-25T17:40:31.468 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.467+0000 7f7d81169640 1 -- 172.21.15.96:0/794730886 learned_addr learned my addr 172.21.15.96:0/794730886 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:31.485 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.484+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260917185 0 0) 0x7f7d7c0af940 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.485 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.485+0000 7f7d73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.485+0000 7f7d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.485+0000 7f7d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.485+0000 7f7d73fff640 10 monclient(hunting): my global_id is 4746 2024-03-25T17:40:31.486 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.485+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d68003520 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245684912 0 0) 0x7f7d7c0b0d50 con 0x7f7d7c14be90 2024-03-25T17:40:31.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.487 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 10 monclient(hunting): my global_id is 4727 2024-03-25T17:40:31.488 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.486+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c0af940 con 0x7f7d7c14be90 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580815042 0 0) 0x7f7d7c0b0a70 con 0x7f7d7c0b1730 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 10 monclient(hunting): my global_id is 4723 2024-03-25T17:40:31.503 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.502+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c0b0d50 con 0x7f7d7c0b1730 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899581059 0 0) 0x7f7d68003520 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 >> v1:172.21.15.96:6790/0 conn(0x7f7d7c14be90 legacy=0x7f7d7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 >> v1:172.21.15.203:6789/0 conn(0x7f7d7c0b1730 legacy=0x7f7d7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 monclient: found mon.noname-a 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.96:6789/0 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c0b1030 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: _finish_auth 0 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d833cd640 5 monclient: authenticate success, global_id 4746 2024-03-25T17:40:31.510 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7d78001690 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d833cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.508+0000 7f7d73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.509+0000 7f7d73fff640 10 monclient: dump: 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.511 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.509+0000 7f7d833cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.0 v1:172.21.15.96:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d780017f0 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 1 -- 172.21.15.96:0/794730886 <== mon.0 v1:172.21.15.96:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7d78001c00 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d833cd640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d833cd640 4 set_mon_vals no callback set 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.515 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.514+0000 7f7d73fff640 10 monclient: dump: 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:31.516 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.518 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 10 monclient: shutdown 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 1 -- 172.21.15.96:0/794730886 >> v1:172.21.15.96:6789/0 conn(0x7f7d7c0b4d60 legacy=0x7f7d7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 1 -- 172.21.15.96:0/794730886 shutdown_connections 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 1 -- 172.21.15.96:0/794730886 >> 172.21.15.96:0/794730886 conn(0x7f7d7c0617a0 msgr2=0x7f7d7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 1 -- 172.21.15.96:0/794730886 shutdown_connections 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.515+0000 7f7d833cd640 1 -- 172.21.15.96:0/794730886 wait complete. 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.516+0000 7f7d833cd640 10 monclient: build_initial_monmap 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.516+0000 7f7d833cd640 10 monclient: monmap: 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:epoch 0 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:last_changed 0.000000 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:created 0.000000 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:31.519 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 1 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.noname-a 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.noname-c 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.516+0000 7f7d833cd640 1 librados: starting msgr at 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.516+0000 7f7d833cd640 1 librados: starting objecter 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.516+0000 7f7d833cd640 1 Processor -- start 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 -- start start 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 librados: setting wanted keys 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 librados: calling monclient init 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient: init 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:31.520 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c0b1730 addr v1:172.21.15.203:6789/0 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c0b4d60 addr v1:172.21.15.96:6790/0 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c14be90 addr v1:172.21.15.96:6789/0 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c136540 con 0x7f7d7c14be90 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c138360 con 0x7f7d7c0b1730 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 1 -- --> v1:172.21.15.96:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7c137300 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.517+0000 7f7d833cd640 10 monclient(hunting): authenticate will time out at 5989.799799s 2024-03-25T17:40:31.521 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.519+0000 7f7d81169640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7d7c0b1730 0x7f7d7c136bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.96:42180/0 (socket says 172.21.15.96:42180) 2024-03-25T17:40:31.522 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.519+0000 7f7d81169640 1 -- 172.21.15.96:0/3523467241 learned_addr learned my addr 172.21.15.96:0/3523467241 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-03-25T17:40:31.522 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.521+0000 7f7d71ffb640 10 client.?.objecter ms_handle_connect 0x7f7d7c0b1730 2024-03-25T17:40:31.522 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.521+0000 7f7d71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.524+0000 7f7d71ffb640 10 client.?.objecter ms_handle_connect 0x7f7d7c0b4d60 2024-03-25T17:40:31.524 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.524+0000 7f7d71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656806078 0 0) 0x7f7d7c138360 con 0x7f7d7c0b1730 2024-03-25T17:40:31.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.526 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.527 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 10 monclient(hunting): my global_id is 4726 2024-03-25T17:40:31.527 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.526+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d60004eb0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.2 v1:172.21.15.96:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629519313 0 0) 0x7f7d7c137300 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 10 monclient(hunting): my global_id is 4730 2024-03-25T17:40:31.529 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.528+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.96:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c138360 con 0x7f7d7c0b4d60 2024-03-25T17:40:31.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.534+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4268167157 0 0) 0x7f7d60004eb0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.534+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d7c137300 con 0x7f7d7c0b1730 2024-03-25T17:40:31.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.535+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7d78001bc0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.535+0000 7f7d71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.535 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.535+0000 7f7d71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.535+0000 7f7d71ffb640 10 monclient(hunting): dump: 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.536 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.535+0000 7f7d71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:31.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.536+0000 7f7d71ffb640 10 client.?.objecter ms_handle_connect 0x7f7d7c14be90 2024-03-25T17:40:31.537 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.536+0000 7f7d71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:31.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 551801759 0 0) 0x7f7d7c137300 con 0x7f7d7c0b1730 2024-03-25T17:40:31.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 >> v1:172.21.15.96:6790/0 conn(0x7f7d7c0b4d60 legacy=0x7f7d7c1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 >> v1:172.21.15.96:6789/0 conn(0x7f7d7c14be90 legacy=0x7f7d7c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:31.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:40:31.541 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 1 monclient: found mon.b 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c1d8450 con 0x7f7d7c0b1730 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 10 monclient: _finish_auth 0 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 5 monclient: authenticate success, global_id 4726 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 10 monclient: _renew_subs 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7c143900 con 0x7f7d7c0b1730 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 10 monclient: _renew_subs 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7c1361c0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.542 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.540+0000 7f7d833cd640 1 librados: init done 2024-03-25T17:40:31.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.541+0000 7f7d833cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.541+0000 7f7d833cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.541+0000 7f7d833cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:31.543 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.541+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7c1401f0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.545+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d780011f0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.545+0000 7f7d71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:31.546 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.545+0000 7f7d737fe640 4 set_mon_vals no callback set 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.826+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2627467344 0 0) 0x7f7d78004cb0 con 0x7f7d7c0b1730 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.826+0000 7f7d71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.826+0000 7f7d71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.826+0000 7f7d71ffb640 10 monclient: dump: 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:epoch 1 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:fsid 7a945e62-7593-49b6-a8a4-685c9e0f96b4 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:last_changed 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.827 INFO:teuthology.misc.health.smithi096.stderr:created 2024-03-25T17:28:02.808273+0000 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:election_strategy: 3 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:0: v1:172.21.15.96:6789/0 mon.a 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:2: v1:172.21.15.96:6790/0 mon.c 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr: 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.827+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2499972302 0 0) 0x7f7d7801ad00 con 0x7f7d7c0b1730 2024-03-25T17:40:31.828 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.828+0000 7f7d80968640 10 monclient: get_auth_request con 0x7f7d6006dc20 auth_method 0 2024-03-25T17:40:31.829 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.828+0000 7f7d71ffb640 10 client.4726.objecter ms_handle_connect 0x7f7d6006dc20 2024-03-25T17:40:31.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.831+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1203174609 0 0) 0x7f7d78054a00 con 0x7f7d7c0b1730 2024-03-25T17:40:31.831 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.831+0000 7f7d71ffb640 10 client.4726.objecter ms_dispatch 0x7f7d7c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T17:40:31.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.831+0000 7f7d71ffb640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T17:40:31.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.831+0000 7f7d71ffb640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T17:40:31.832 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.831+0000 7f7d71ffb640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T17:40:31.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.834+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d78050340 con 0x7f7d7c0b1730 2024-03-25T17:40:31.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.834+0000 7f7d71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:31.834 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:31.834+0000 7f7d71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:32.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.056+0000 7f7d833cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.056+0000 7f7d833cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.057 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.056+0000 7f7d833cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T17:40:32.058 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.056+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d7c0627b0 con 0x7f7d7c0b1730 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.076+0000 7f7d71ffb640 1 -- 172.21.15.96:0/3523467241 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 590302222) 0x7f7d78052440 con 0x7f7d7c0b1730 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.076+0000 7f7d71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.076+0000 7f7d71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi096.stdout: 2024-03-25T17:40:32.077 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"7a945e62-7593-49b6-a8a4-685c9e0f96b4","created":"2024-03-25T17:29:02.810080+0000","modified":"2024-03-25T17:39:06.920771+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:06.920771+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67c12af-45d9-4991-a443-4e6b4ce66e07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ad03c452-1e7a-4d1f-bbd6-b3c737a5e354","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e5d86866-0c9f-4715-90ed-4761036cb0b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"318e04fa-02d7-4358-879a-2613abedef73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"733daa39-6ad9-4192-b243-b61c78d9c006","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5cc3897-7dcb-44b1-b07b-66d0ccc18405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"154501bc-27c0-44aa-b485-1c14c8037882","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"76585eac-e095-4e42-acf6-482d6dbad102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.078+0000 7f7d833cd640 10 librados: watch_flush enter 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.078+0000 7f7d833cd640 10 librados: watch_flush exit 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.078+0000 7f7d833cd640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.078+0000 7f7d833cd640 10 client.4726.objecter successfully canceled tick 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 >> v1:172.21.15.203:6800/3611964061 conn(0x7f7d6006dc20 legacy=0x7f7d600700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 10 monclient: shutdown 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 >> v1:172.21.15.203:6789/0 conn(0x7f7d7c0b1730 legacy=0x7f7d7c136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:32.080 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 shutdown_connections 2024-03-25T17:40:32.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 >> 172.21.15.96:0/3523467241 conn(0x7f7d7c0617a0 msgr2=0x7f7d7c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:32.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 shutdown_connections 2024-03-25T17:40:32.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 -- 172.21.15.96:0/3523467241 wait complete. 2024-03-25T17:40:32.081 INFO:teuthology.misc.health.smithi096.stderr:2024-03-25T17:40:32.079+0000 7f7d833cd640 1 librados: shutdown 2024-03-25T17:40:32.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:32.092 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T17:40:32.094 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T17:40:32.094 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T17:40:32.094 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:32.213 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:56.440 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.271 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:56.498 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.308 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:56.524 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.309 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:11:56.535 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.319 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:56.535 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.342 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:56.568 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.347 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:56.573 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.353 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:56.569 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:32.385 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:11:56.611 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:33.967 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:11:25.715 35500== 2024-03-25T17:40:33.967 INFO:tasks.ceph.osd.0.smithi096.stderr:==00:00:11:25.715 35500== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:34.003 INFO:tasks.ceph.osd.0.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:40:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:40:38.198 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T17:40:38.198 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T17:40:38.199 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T17:40:38.199 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:40.102 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:11:31.850 35501== 2024-03-25T17:40:40.102 INFO:tasks.ceph.osd.1.smithi096.stderr:==00:00:11:31.850 35501== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:40.143 INFO:tasks.ceph.osd.1.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:40:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:40:44.300 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T17:40:44.301 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T17:40:44.301 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T17:40:44.301 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:46.137 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:11:37.865 35503== 2024-03-25T17:40:46.137 INFO:tasks.ceph.osd.2.smithi096.stderr:==00:00:11:37.865 35503== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:46.182 INFO:tasks.ceph.osd.2.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:40:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:40:50.403 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T17:40:50.404 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T17:40:50.404 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T17:40:50.404 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:52.135 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:11:43.873 35502== 2024-03-25T17:40:52.136 INFO:tasks.ceph.osd.3.smithi096.stderr:==00:00:11:43.873 35502== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:52.177 INFO:tasks.ceph.osd.3.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:40:52.301 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:16.527 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.311 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:16.537 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.330 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:16.556 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.359 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:16.585 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.360 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:16.576 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.362 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:16.589 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.364 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:16.579 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.382 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:16.608 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:52.390 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:16.605 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:40:56.506 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T17:40:56.506 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T17:40:56.507 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T17:40:56.507 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:58.600 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:11:50.345 35576== 2024-03-25T17:40:58.601 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:11:50.345 35576== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:58.635 INFO:tasks.ceph.osd.4.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:02.608 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T17:41:02.609 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T17:41:02.609 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T17:41:02.609 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:04.441 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:11:56.179 35577== 2024-03-25T17:41:04.441 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:11:56.179 35577== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:04.479 INFO:tasks.ceph.osd.5.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:08.710 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T17:41:08.711 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T17:41:08.711 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T17:41:08.711 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:10.407 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:12:02.128 35578== 2024-03-25T17:41:10.407 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:12:02.128 35578== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:10.452 INFO:tasks.ceph.osd.6.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:12.338 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:36.564 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.339 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:36.555 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.341 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:36.568 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.344 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:36.559 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.359 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:36.586 35182== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.373 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:36.576 35018== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.388 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:36.615 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.392 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:36.619 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:12.407 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:36.634 35185== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:14.813 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T17:41:14.814 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T17:41:14.814 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T17:41:14.814 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:16.239 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:12:07.960 35579== 2024-03-25T17:41:16.239 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:12:07.960 35579== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:16.269 INFO:tasks.ceph.osd.7.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:20.916 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T17:41:20.917 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T17:41:20.917 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T17:41:20.917 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T17:41:20.917 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:20.943 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T17:41:20.943 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T17:41:20.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:20.970 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T17:41:20.971 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T17:41:20.971 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T17:41:20.971 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:21.001 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-03-25T17:41:20.987+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35153) UID: 0 2024-03-25T17:41:21.001 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-03-25T17:41:20.989+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T17:41:21.405 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:45.632 35182== 2024-03-25T17:41:21.405 INFO:tasks.ceph.mon.a.smithi096.stderr:==00:00:12:45.632 35182== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:21.414 INFO:tasks.ceph.mon.a.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:27.072 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T17:41:27.073 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T17:41:27.073 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T17:41:27.073 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:27.090 INFO:tasks.ceph.mon.c.smithi096.stderr:2024-03-25T17:41:27.078+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35155) UID: 0 2024-03-25T17:41:27.091 INFO:tasks.ceph.mon.c.smithi096.stderr:2024-03-25T17:41:27.079+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T17:41:27.592 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:51.819 35185== 2024-03-25T17:41:27.592 INFO:tasks.ceph.mon.c.smithi096.stderr:==00:00:12:51.819 35185== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:27.602 INFO:tasks.ceph.mon.c.smithi096.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:33.176 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi096 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T17:41:33.176 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T17:41:33.176 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T17:41:33.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:33.194 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-03-25T17:41:33.180+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35014) UID: 0 2024-03-25T17:41:33.194 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-03-25T17:41:33.181+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T17:41:33.621 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:57.837 35018== 2024-03-25T17:41:33.621 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:57.837 35018== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:33.631 INFO:tasks.ceph.mon.b.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:39.278 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T17:41:39.278 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T17:41:39.278 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T17:41:39.279 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T17:41:39.596 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:41:39.597 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T17:41:39.717 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:41:39.717 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T17:41:39.784 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:41:39.784 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T17:41:39.848 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:41:39.849 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T17:41:39.918 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:41:39.918 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T17:41:40.065 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:41:40.065 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T17:41:40.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:41:40.128 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T17:41:40.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:41:40.178 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T17:41:40.227 INFO:tasks.ceph:Archiving mon data... 2024-03-25T17:41:40.229 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/data/mon.a.tgz 2024-03-25T17:41:40.229 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-03-25T17:41:40.246 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.PHDmK6bKv4 2024-03-25T17:41:40.247 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.PHDmK6bKv4 2024-03-25T17:41:40.351 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.PHDmK6bKv4 2024-03-25T17:41:40.437 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.PHDmK6bKv4 is 135KB 2024-03-25T17:41:40.460 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.PHDmK6bKv4 2024-03-25T17:41:40.474 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/data/mon.c.tgz 2024-03-25T17:41:40.474 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-03-25T17:41:40.529 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.0zJrPbWQug 2024-03-25T17:41:40.529 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.0zJrPbWQug 2024-03-25T17:41:40.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.0zJrPbWQug 2024-03-25T17:41:40.700 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.0zJrPbWQug is 135KB 2024-03-25T17:41:40.750 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.0zJrPbWQug 2024-03-25T17:41:40.764 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/data/mon.b.tgz 2024-03-25T17:41:40.765 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-25T17:41:40.781 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.708c3gWZlV 2024-03-25T17:41:40.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.708c3gWZlV 2024-03-25T17:41:40.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.708c3gWZlV 2024-03-25T17:41:40.976 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.708c3gWZlV is 135KB 2024-03-25T17:41:41.027 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.708c3gWZlV 2024-03-25T17:41:41.041 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T17:41:41.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:41:41.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:41:41.430 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:41:41.668 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T17:41:41.671 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi096/crash 2024-03-25T17:41:41.672 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:41:41.699 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi203/crash 2024-03-25T17:41:41.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:41:41.725 INFO:tasks.ceph:Compressing logs... 2024-03-25T17:41:41.726 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:41:41.743 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35174.log 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35174.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35174.log.gz 2024-03-25T17:41:41.812 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35430.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35265.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35265.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35430.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35346.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35265.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35219.log 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-03-25T17:41:41.813 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35219.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.35386.log -- replaced with /var/log/ceph/ceph-client.admin.35219.log.gz 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35305.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35386.log.gz 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr: /var/log/ceph/ceph.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35305.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35305.log.gz 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35477.log 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:41:41.814 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35477.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35477.log.gz 2024-03-25T17:41:41.818 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40504.log 2024-03-25T17:41:41.818 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-03-25T17:41:41.818 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40816.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40504.log: /var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40504.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.42106.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40816.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.35195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40309.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39919.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40309.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40660.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39841.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39919.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40309.log.gz 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2024-03-25T17:41:41.819 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39919.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40660.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40738.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39841.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39294.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T17:41:41.820 INFO:teuthology.orchestra.run.smithi096.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39997.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41402.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39997.log: /var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41011.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39997.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39607.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41402.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41638.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41011.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41363.log 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-03-25T17:41:41.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39607.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41638.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41128.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41363.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41285.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39646.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39685.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39568.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39763.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41285.log.gz 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39568.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.39685.log.gz /var/log/ceph/ceph-client.admin.40153.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41519.log 2024-03-25T17:41:41.822 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39568.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41989.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39763.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40153.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41989.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41519.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35526.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40894.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41989.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.35526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29330.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35526.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39056.log: /var/log/ceph/ceph.tmp-client.admin.29330.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29330.log.gz 2024-03-25T17:41:41.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39724.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41441.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39724.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39958.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41441.log.gz 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40855.log 2024-03-25T17:41:41.824 INFO:teuthology.orchestra.run.smithi096.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39958.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42028.log: /var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39333.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40855.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40855.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40426.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39333.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40777.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41246.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40777.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40426.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40075.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41246.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40777.log.gz 2024-03-25T17:41:41.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40348.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41246.log.gzgzip 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41872.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40075.log.gz 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42067.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr: 86.3%/var/log/ceph/ceph-client.admin.39097.log: -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42067.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41872.log.gz 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41755.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40972.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42067.log.gz 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36562.log 2024-03-25T17:41:41.826 INFO:teuthology.orchestra.run.smithi096.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41168.log 86.9% 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.36562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39017.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41168.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41755.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36562.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39017.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41168.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35673.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41050.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39017.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-osd.3.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.35673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42225.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40699.log 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35673.log.gz 2024-03-25T17:41:41.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39880.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40582.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42225.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35499.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40699.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5/var/log/ceph/ceph-client.admin.40582.log: --verbose -- /var/log/ceph/ceph-client.admin.39490.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39880.log.gz/var/log/ceph/ceph-client.admin.35499.log: 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41833.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40582.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42145.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35499.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41833.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39490.log.gz 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42184.log 2024-03-25T17:41:41.828 INFO:teuthology.orchestra.run.smithi096.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41833.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39802.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42145.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39411.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39802.log: gzip -5 --verbose -- 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42184.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.40270.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39411.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39802.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39451.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40270.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39411.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39529.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39451.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40270.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40192.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39529.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39451.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40192.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39529.log.gz 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-03-25T17:41:41.829 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.0.log: 86.3%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40192.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41950.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose/var/log/ceph/ceph.log: -- /var/log/ceph/ceph-client.admin.40387.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41911.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40387.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40387.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40933.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37638.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41911.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40933.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40933.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.40114.log 2024-03-25T17:41:41.830 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.admin.40543.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40465.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40114.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41089.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.40465.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40114.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40465.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.41716.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.39372.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41716.log: /var/log/ceph/ceph-client.admin.41089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41560.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.39372.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41716.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41089.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42264.log 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-03-25T17:41:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.41560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38805.log 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42264.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41324.log 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41560.log.gz 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42264.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.38805.log: /var/log/ceph/ceph-client.admin.41324.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42303.log: 86.3% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38805.log.gz 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41324.log.gz 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.40231.log 2024-03-25T17:41:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40231.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2024-03-25T17:41:41.856 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T17:41:42.608 INFO:teuthology.orchestra.run.smithi096.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T17:41:42.706 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T17:41:42.726 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T17:41:42.741 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T17:41:42.917 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T17:41:42.919 INFO:teuthology.orchestra.run.smithi096.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T17:41:42.981 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T17:41:42.998 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T17:41:46.643 INFO:teuthology.orchestra.run.smithi096.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T17:41:47.101 INFO:teuthology.orchestra.run.smithi203.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T17:41:47.103 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T17:41:47.103 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m5.316s 2024-03-25T17:41:47.103 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m9.000s 2024-03-25T17:41:47.103 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.324s 2024-03-25T17:41:47.202 INFO:teuthology.orchestra.run.smithi096.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T17:41:47.204 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-25T17:41:47.204 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m5.449s 2024-03-25T17:41:47.204 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m14.148s 2024-03-25T17:41:47.204 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.535s 2024-03-25T17:41:47.204 INFO:tasks.ceph:Archiving logs... 2024-03-25T17:41:47.204 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi096/log 2024-03-25T17:41:47.206 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:41:49.614 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi203/log 2024-03-25T17:41:49.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:41:51.129 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:41:51.336 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3b1259d0f695464e861db9dee64a583b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:41:51.340 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T17:41:51.349 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T17:41:51.358 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T17:41:51.358 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:41:51.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:41:51.448 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T17:41:51.512 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T17:41:51.513 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:41:51.516 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:41:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:41:51.532 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:41:51.535 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:41:51.535 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-03-25T17:41:51.536 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +775us[ +775us] +/- 73ms 2024-03-25T17:41:51.536 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -1253us[-1228us] +/- 57ms 2024-03-25T17:41:51.536 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1086us[+1086us] +/- 62ms 2024-03-25T17:41:51.536 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:41:51.537 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:41:51.537 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-25T17:41:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +773us[ +788us] +/- 73ms 2024-03-25T17:41:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1250us[-1236us] +/- 57ms 2024-03-25T17:41:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +1083us[+1083us] +/- 62ms 2024-03-25T17:41:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:41:51.539 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T17:41:51.547 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T17:41:51.548 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T17:41:51.556 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:41:51.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:41:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:41:51.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:41:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:41:51.782 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T17:41:51.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T17:41:51.800 INFO:teuthology.task.internal:Duration was 5740.121510 seconds 2024-03-25T17:41:51.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T17:41:51.810 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T17:41:51.810 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:41:51.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:41:51.854 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:41:51.868 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:41:52.244 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T17:41:52.244 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-03-25T17:41:52.245 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:41:52.305 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:41:52.305 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:41:52.330 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T17:41:52.331 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:41:52.348 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:41:52.402 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T17:41:52.402 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:41:52.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:41:52.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T17:41:52.556 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T17:41:52.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:41:52.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:41:52.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T17:41:52.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:41:52.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:41:52.648 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-03-25T17:41:52.675 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-25T17:41:52.723 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:41:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:41:52.737 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:41:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:41:52.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T17:41:52.788 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T17:41:52.789 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi096 2024-03-25T17:41:52.789 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:41:52.824 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240/remote/smithi203 2024-03-25T17:41:52.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:41:52.859 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T17:41:52.859 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:41:52.861 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:41:52.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T17:41:52.919 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T17:41:52.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T17:41:52.929 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T17:41:52.929 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:41:52.932 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:41:52.944 INFO:teuthology.orchestra.run.smithi096.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:41 /home/ubuntu/cephtest 2024-03-25T17:41:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: 265504 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:41 /home/ubuntu/cephtest 2024-03-25T17:41:52.964 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T17:41:52.974 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T17:41:52.994 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T17:41:53.020 DEBUG:teuthology.nuke:shortname: smithi096 2024-03-25T17:41:53.020 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:41:53.046 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:48.630761', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGhOWy56aoh5UtNOg3LmvNGF8b0GwQ4bUk7rw2EVB84w1QfE8UxSmJcQHHMGLXeWpLlkJ1tM67+kxnDCBO/sQI='} 2024-03-25T17:41:53.050 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-25T17:41:53.050 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:41:53.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:48.632359', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJOvNpSlpcxMfV8QgjnjUklGa74zPD66ymf4Wxg95gUb/NB/Bhwumx03H/2i9zqOSV3uA5AUxbS8D+tIIdcOwm8='} 2024-03-25T17:41:53.095 INFO:teuthology.orchestra.console.smithi096:Power off 2024-03-25T17:41:53.095 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:41:53.117 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-03-25T17:41:53.118 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:41:53.136 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-25T17:41:53.136 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:41:53.147 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-03-25T17:41:53.160 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-25T17:41:53.160 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:41:53.185 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T17:41:57.148 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:41:57.186 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:41:57.260 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-03-25T17:41:57.297 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T17:42:01.265 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:01.298 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:01.376 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-03-25T17:42:01.408 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T17:42:01.477 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-03-25T17:42:05.412 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:05.524 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-25T17:42:05.624 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-25T17:42:05.730 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5740.1215097904205 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3b1259d0f695464e861db9dee64a583b status: fail success: false 2024-03-25T17:42:05.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:42:05.811 INFO:teuthology.run:FAIL